Trial By Error: Dutch CBT Study for Long Covid Proves that Unblinded Studies with Subjective Outcomes Generate Positive Reports

By David Tuller, DrPH

Three years ago, I wrote a blog post about a problematic Dutch study that had been funded by a major health agency and was being led by Hans Knoop, a professor of medical psychology at Amsterdam University Medical Centers. The study sought to test whether a course of cognitive behavior therapy starting months after a bout of acute Covid-19—rather than years later— could reduce levels of reported fatigue and prevent it from becoming chronic.

Professor Knoop is a long-time colleague of the authors of the now-discredited PACE trial. In a Lancet commentary accompanying the publication of the PACE results, he and a colleague declared that many patients had met “a strict criterion for recovery”—a ridiculous and untrue claim. 

This new study—like so much of the research from Professor Knoop and his colleagues in the world of psychosomatic medicine—was unblinded and relied on subjective, self-reported outcomes. This design is fraught with potential bias. Given the design, the study was destined to produce positive results on these subjective outcomes—and now, not surprisingly, it has, with the results published in the journal Clinical Infectious Diseases

Also not surprisingly, the results for the one objective outcome included in the protocol—actigraphy to measure levels of physical activity at baseline and right after treatment—were not reported or mentioned in the paper. Professor Knoop has deployed this strategy before—most recently in 2017 when he and colleagues published positive subjective outcomes but failed to report null actometer results in a study of CBT for treating fatigue after Q-fever; these null results were published two years later and ignored. And in a similar fashion a dozen years ago, Professor Knoop and several colleagues buried disappointing actigraphy results from three trials of CBT for ME/CFS.

Twitter threads from @lucibee and @anilvanderzee point out some of the major issues with the new study, called “Efficacy of cognitive behavioral therapy targeting severe fatigue following COVID-19: the results of a randomized controlled trial.” The study was also the subject of a lively discussion on the Science For ME forum. The CBT course, called “Fit After Covid,” included online modules along with in-person or online contact with a therapist.

The study’s 114 participants were all suffering from what was identified as severe fatigue three to 12 months after their acute infections. They were randomized into a group receiving the CBT  program and group receiving care as usual (CAU)—a design that undermines the claim in the article title that the study was “controlled.” 

Yes, there was a comparison arm. But the mean number of interactions between therapist and patient in the CBT arm was almost 12, and the study did not offer members in the CAU group a parallel amount of time and attention.

If participants know they are receiving an active treatment in a trial—a full course of therapeutic encouragement, for example—and they are told that this treatment has been found to be successful in other circumstances, it stands to reason they would be more likely to report benefits than people who know they did not receive the possibly helpful treatment. The authors mention this imbalance between the groups as a limitation but nonetheless still call the trial “controlled,” even though they are not controlling for this important factor.


CBT targets seven domains of thought and activity

The trial was based on a “cognitive-behavioral model” and the CBT was specifically designed to target seven perceived domains that could perpetuate the fatigue. These were: a disrupted sleep-wake pattern, unhelpful beliefs about fatigue, a low or unevenly distributed activity level, perceived low social support, problems with psychological processing of COVID-19, fears and worries regarding COVID-19, and poor coping with pain. The CBT group received the intervention for 17 weeks. 

Let’s be clear. Everyone who has been sick could benefit from someone—a good social worker, a life coach, grandma, a counselor, or a CBT therapist—offering them common sense advice about sleep and activity levels and the need to find more friends or call their siblings if they’re feeling low. Of course it is useful to help people address fears and worries about the pandemic. All these things are likely to make them feel better, emotionally and physically, especially compared to people who are not offered anything comparable. 

And of course that will make them more likely to answer more positively in general on questionnaires, including fatigue questionnaires. That doesn’t mean you’re treating anyone’s fatigue—just that you’re providing human and social support of the kind we all could benefit from in tough times. It should be expected that this would be reflected in modest improvements in questionnaire scores—especially in an open label trial in which people know they are getting a “treatment” that they believe could help them get better.

The authors acknowledged this limitation but noted that members of the CAU group were able to access other treatments that could have helped provide some balance to the experiences of the separate arms. The paper noted, for example, that “most participants in the CAU group members of the CAU group received care, including exercise.” 

Uh, oh! The study makes no mention of post-exertional malaise (PEM), a core characteristic of ME/CFS and in many cases of long Covid. If some CAU group members had PEM and were being encouraged to exercise, that could explain why they reported worse fatigue at the end. It might also explain why the CBT group had fewer reported adverse events than the CAU group.

Besides fatigue, the participants were required as an entry requirement to demonstrate limitations in physical function by a low score on a questionnaire and/or limitations in “social functioning” on a different scale. This raises the possibility that some were suffering from primary depression rather than being physically disabled. Any such patients might well have benefited from a course of CBT, which is frequently prescribed treatment for depression.

The trial’s primary outcome was the difference in the means between the two groups right after treatment and six months later on the fatigue severity sub-scale of the Checklist Individual Strength (CIS), a 20-item fatigue questionnaire. The fatigue severity sub-scale has eight items, with each one rated on a 1-7 scale, with higher scores indicating greater fatigue. The total score ranges from a low of eight to a hight of 52. 


No objective findings but modest subjective results

The study showed—wow!—that those who received the intervention said they were less tired than those that didn’t get the intervention. The overall mean for the CBT group’s CIS scores right after treatment and six months later was 8.8 points lower than the the mean for the CAU group. This is not a huge gap on a 52-point scale. The mean score for the CBT group at six months—31.5—still represents significant fatigue. The difference between the groups seems well within what one might expect from any bias stemming from the study design.

Secondary measures—all subjective—also favored the intervention.

But the absence of the actigraphy results casts doubt on even these modest reported subjective benefits. The actigraphy results would have revealed participants’ objective levels of physical activity. The protocol called for participants to wear actigraphs around their wrists to monitor their activity for 14 days, both at the start of the trial and right at the end of the course of therapy. (For unexplained reasons, the protocol did not call for actigraphy at the six-month post-therapy time point.)

Beyond not providing the results, the authors didn’t explain why they decided not to provide them. This omission of salient data aligns with past practice by Dutch investigators in this domain—including Professor Knoop. Besides the 2017 Q-fever study, in three previous Dutch studies for psycho-behavioral interventions for ME/CFS that included actigraphy as an outcome, the authors published the positive subjective outcomes in the initial papers but left out the objective outcome. 

Finally, years later, they published the actigraphy results from all three papers in a single article. The actigraphy had null results in all three studies. “Although CBT effectively reduced fatigue, it did not change the level of physical activity,” concluded the authors, including Professor Knoop. In other words, the CBT appeared to improve reporting on fatigue questionnaires but did not lead to any real change in how much people do physically. This apparent conflict between subjective and objective outcomes did not shake the authors’ confidence in the accuracy of the former.

In this case, it is hard to imagine that Professor Knoop would have omitted the actigraphy results from the published report had they documented that CBT conferred objective measurable benefits. What reason would there be? But if they showed no change in the CBT group, or worse, and Professor Knoop publishes those data two years from now, no one will notice. The study has already been published with modestly positive subjective findings, and that’s already sucked up the attention. 

7 thoughts on “Trial By Error: Dutch CBT Study for Long Covid Proves that Unblinded Studies with Subjective Outcomes Generate Positive Reports”

  1. David wrote “grandma” and it reminded me of this - . I suspect that there’s nobody quite like a therapist to make people feel momentarily in the pink when they’re not….. (except Grandma perhaps?)

  2. Ellen Goudsmit

    If they did actigraphy, get the report of the study sent to the funders. That is how we found there was no change in Prins et al. I passed the results onto the Lancet, which, surprise surprise, rejected my letter. Ignorance is bliss.

    Not all with long covid feel tired. Some have damaged lungs. Were they included? All potential reasons for fatigue noted and people placed in subsets?

  3. I get that it was a multicenter trial in the Netherlands that might have required quite a few people to be involved. But I do wonder why it needed authors from Australia. Were the people from the Netherlands lacking knowledge or skills or something?

  4. I wonder if patients were involved in the design of the trial and whether or not this was a patient research priority. I also wonder about the level of qualifications of the CBT trainers since there was a pre Covid trial I heard about (for various symptoms which were being described as ‘medically unexplained’ but were mostly those experienced by people with balance/motion sensation conditions) where the CBT outcomes were considerably better when the trainer had nursing qualifications. Unless the training for the trainers was standardised in this trial (which would be hard to ascertain) the outcomes could be impacted by that, too.

  5. David wrote: “The authors mention this imbalance between the groups as a limitation…”

    What often seems to happen is that the limitations of a study will stay with the paper and are not transferred across when the study is subsequently reported on, but limitations should not, (in my opinion anyway), be regarded as some sort of tick box exercise to get a paper through peer review. Can anyone spot where the limitations of the study are mentioned in these reports - and ?

  6. Mike Fraumeni

    One thing I’m still learning, and this is well known in the biomedical literature, is that physical activity is not a panacea for all diseases or conditions or illnesses or symptoms. Myself as someone who studied physical education in university and graduated with an Hon. B.Arts majoring in physical education, I have a bias that I continue to deconstruct although still not doing this well at all times as I recently learned from the ME/CFS community the hard way, that physical activity is simply good for everyone at all times despite ones health conditions. This is a misinformed perception I’m realizing. I also realize that as someone who doesn’t have ME/CFS, I really have no true understanding of this disease and I probably never will as long as I don’t seem to have it. However, I will continue to try and change my inherent bias through reading and learning more about ME/CFS. And I do apologize to anyone I’ve offended over the years as a result of my bias/ignorance.

Comments are closed.

Scroll to Top