Trial By Error: REGAIN Team Responds to Criticism with Unconvincing Defenses of Methodological Missteps

By David Tuller, DrPH

In recent weeks, I have been urging The BMJ to correct a flawed University of Warwick trial of an online mental and physical health rehab program for people with prolonged symptoms at least three months after hospitalization for Covid-19. The primary outcome was health-related quality of life, assessed with a measure called the PROPr score. Most recently, I sent a letter signed by two dozen scientists, clinicians and academics about the issue. (I first wrote about the study here; an earlier letter to the journal is here.)

Now the authors of the REGAIN trial have posted a rapid response in an effort to rebut the criticisms. Their response suggests a poor understanding of the important role that caution and humility should play in the reporting of scientific studies. The REGAIN team has justified its maximalist presentation of its findings by offering unsatisfactory and inadequate explanations for its decision-making.

The trial was unblinded and relied solely on subjective outcomes—a study design that generates an unknown amount of bias. Authors of such studies usually fail to include sufficient caveats that very modest reports of improvement are likely to be attributable to placebo effects rather than the impact of the intervention itself.

Beyond that basic issue, we expressed two major concerns. First, we were troubled that the authors extrapolated their findings to the entire Long Covid population and not just to those who had been hospitalized. Second, we questioned the decision to call the intervention “clinically effective” even though the results did not reach the currently recommended threshold for a minimal clinically important difference (MCID) for the PROPr score.

Yesterday, I was informed by the handling editor for the paper, Dr Nazrul Islam, that the REGAIN team had posted their rebuttal in a rapid response. A short time later, I sent him an e-mail addressing the first extrapolation issue and explaining why the effort to brush it off did not meet the smell test—or at least my smell test.

Here is the letter:

Dear Dr Islam–

Thank you for the update. We will review the response in detail, but I will say from a first glance that it is too clever by half and clearly skates over key criticisms. Just one glaring example: In their effort to rebut the argument that they have extrapolated the findings way beyond the population studied, the authors’ reference the conclusion of the full paper rather than the conclusion of the abstract. 

In our letter, we referenced the abstract, which reads: “In adults with post-covid-19 condition, an online, home based, supervised, group physical and mental health rehabilitation programme was clinically effective at improving health related quality of life at three and 12 months compared with usual care.” Furthermore, the box highlighting “What is already. known on this topic” and “What this study adds” includes zero mention of the highly salient detail that these patients had been hospitalized. As you undoubtedly know, readers are far more likely to review the abstract and highlights sections than they are to read the paper to the very end.

Had McGregor et al added “at least three months after hospital discharge for covid-19” to the sentence in the abstract and to the box highlighting the findings, we would not have raised this particular issue. Apparently, McGregor et al and The BMJ’s editorial team believe it is entirely appropriate for investigators to offer more sweeping statements in the conclusions in a paper’s abstract and in its highlights section as long as the very last paragraph of the paper’s text includes more accurate information. 

We strongly disagree. Such flagrant internal contradictions are not acceptable in reporting scientific findings. First-year epidemiology students at Berkeley would be reprimanded for this transparent effort to convey the impression in the paper’s most high-profile sections that the findings are broadly applicable to the entire population of patients with post-covid-19 condition rather than limited to the much smaller cohort of those who were hospitalized. 

The responses to our other concerns are equally disingenuous and inadequate–more on that to come. Readers of The BMJ deserve better.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley 

**********

Pathetic defense for conflicting accounts of the primary outcome’s threshold for “minimal clinically important difference”

As to our second major point, the authors write that the criticism of is “a reprise of material from our discussion where we sought to put the observed effect size in context.” Actually, the criticism was prompted by the fact that the reference cited by the authors contradicts their explicit and declarative claim in the Methods section that the accepted MCID for the PROPr score is the same as for other preference-based measures. This is the sentence in question: “As with other preference based measures such as the EuroQol 5 dimension 5 level (EQ-5D-5L) instrument, a difference of 0.03 to 0.05 is considered to be clinically important.”

However, this statement is untrue. The reference for it is an authoritative “frequently asked questions” page for the PROPr measure. This page does confirm that 0.03 to 0.05 is the standard MCID range for the category of preference-based measures to which PROPr belongs. But the FAQ does not support the assertion in the Methods section that this range currently still applies to the PROPr score. It clearly notes that “we currently recommend using 0.04” as the MCID threshold; moreover, it adds that a “conservative” MCID would be twice as big—a difference of 0.08. Even though this recommendation derives from “work-in-progress” and has not been “formally evaluated,” as the FAQ explains, it is nonetheless the current recommendation.

The REGAIN team mentions this highly salient nuance only in the Discussion section; the authors appear to want to be congratulated for what they seem to think is a commendable act of transparency. But as with the issue of appropriate or inappropriate extrapolation, the authors and The BMJ editorial team clearly believe it is acceptable to offer false or at least incomplete and misleading statements early on in a paper as long as the actual facts are provided deep in the bowels of the text. This is a cynical and deceptive approach to the reporting of scientific results.

Investigators with integrity tend to be modest rather than expansive in their interpretations of their data. It would be refreshing if those studying rehab programs for complex illnesses like ME/CFS and Long Covid curbed their evident enthusiasm for hyping findings as significantly more impressive than they are. But that will never happen as long as editors at major journals like The BMJ are so willing to overlook egregious methodological missteps and publish what is essentially propaganda.