Trial By Error: Article in New Scientist Questions Value of Exercise for Long COVID (and quotes me)

By David Tuller, DrPH

*This is a crowdfunding month at UC Berkeley. If you’d like to support my work, the link is here.

**********

When it comes to exercise and Long COVID, investigators have routinely claimed success for their interventions–and many news articles have followed their lead in covering the issue. Most reporters don’t have the time or bandwidth to look more deeply and challenge that narrative—in particular, by checking out whether the much-touted trials really live up to their hype. 

A story this week in New Scientist, a London-based publication, breaks this mold. Journalist Alexandra Thompson did her homework. She had approached me a few months ago, indicating that she was working on a piece about how exercise was emerging as a possible treatment for Long COVID. I pointed out that it might appear from the claims of investigators and accompanying media reports that exercise was effective, but that the science did not really support this conclusion.

We spoke at length, and I knew she was talking with other critics of this body of research as well as with its proponents. In circumstances like these, you can only hope a story turns out reasonably well.

So I was pleased when I saw the story today with the following headline: “Exercise advice for long covid may be doing more harm than good.” And this subhead: “Exercise has been touted as a tool for managing and treating long covid, but much of the evidence has neglected one of its most debilitating symptoms: post-exertional malaise.” In the story, Thompson quotes, among others, Irish patient advocate and researcher Tom Kindlon; Todd Davenport, a professor of physical therapy at the University of the Pacific in Stockton, California; and me.

Here’s the opening:

“In the hunt for ways to alleviate long covid—a relatively new condition with no cure, experienced by millions of people worldwide after contracting covid-19—exercise has been a bright spot. It’s drug-free, it costs nothing and a handful of studies have suggested it boosts long covid recovery. But concern is growing that these studies aren’t robust enough to support exercise as a treatment approach, reigniting a decade-long controversy over the use of exercise in addressing other conditions, such as chronic fatigue syndrome.”

The story then highlights a Scottish trial from the University of Glasgow, identifying it as “one of the highest-profile studies of exercise in long covid.” The trial, which received positive media attention when it was published last November by JAMA Network Open, purported to prove that a program of “resistance exercise therapy” was helpful. An article disseminated on Yahoo, headlined “Long-COVID Patients Should Focus on This Training,” included the following quote from the study’s lead investigator: “Our study shows the benefits of strength training for recovery after COVID-19 and suggests that people suffering from persistent symptoms after a COVID-19 infection could benefit from this type of training.” 

The primary outcome was a standard measure of exercise capacity called the Incremental Shuttle Walk Test (ISWT), in which subjects start off walking slowly and gradually pick up speed. As evidence of the intervention’s purported effectiveness, the investigators noted that those in the intervention arm improved an average of 83 meters on the ISWT from baseline, compared to an increase of 47 meters for those in the comparison arm.

But the trial protocol had designated 46 meters as the threshold for what was considered a clinically significant benefit, and the statistically adjusted difference of 36.5 meters between the two groups was significantly lower than that. In other words, the results were statistically significant but clinically insignificant. In the real world, the difference between the outcomes in the intervention and comparison arms was too small to make any meaningful difference to patients, or even to be noticeable. per the trial’s own assessment methods.

As I wrote in a blog post at the time:

“Here’s the main problem for the investigators and their primary outcome. Per the statistical analysis plan outlined in their protocol, the ISWT’s threshold for clinical significance—known as the minimal clinically important difference (MCID)—is 46 meters. The investigators cited an authoritative 2022 paper from the European Respiratory Society (ERS)–“Use of exercise testing in the evaluation of interventional efficacy: an official ERS statement”–for this MCID. They used it to informed their power calculations for determining the needed sample size…

“In the trial, participants in the intervention arm performed better than the control group by an average of 36.5 meters. Let’s acknowledge the obvious: 36.5 meters is quite a bit lower than the MCID of 46 meters that the investigators themselves referenced in their statistical analysis plan. Oops!

“Researchers with a robust sense of integrity would have reported such a salient detail, however disappointing or embarrassing or contrary to their expectations. Yet the JAMA Network Open paper made no mention of it, effectively disappearing a very revealing data point. This omission, in and of itself, serves to misrepresent the findings and is arguably a form of research misconduct. Readers deserve to know that the trial’s primary outcome did not reach its own threshold for clinical significance.”

Most readers of the study, or readers of articles about the study, would not know these important facts. To Thompson’s credit, she has drawn attention to them, in the process raising questions about the ubiquitous promotion of exercise for Long COVID. As she writes in the New Scientist article:

“The study quickly garnered press coverage and was widely discussed on social media. But many scientists have pointed out issues with this trial…The difference between the distances walked by the control and exercising groups fell 10 metres short of the minimum threshold for clinical significance chosen by the team at the start of the experiment. ‘If you don’t achieve the level that’s minimally clinically important, you don’t go around claiming success,’ says David Tuller at the University of California, Berkley [sic]. In response, Berry [Colin Berry, the lead investigator] says it’s not for us to say whether an individual would benefit from this improved mobility. ‘I think that’s open to interpretation.’”

Colin Berry’s response doesn’t make a lot of sense. What, exactly, is “open to interpretation”? The facts are the facts–the exercise intervention did not offer participants, on average, clinically significant benefits on the primary outcome, according to the trial’s predesignated metrics. Yet Berry and his colleagues chose not to mention this key detail about the threshold for clinical significance in their published report. The decision to withhold this information was ethically and scientifically indefensible.

The New Scientist article also discusses the failure of Long COVID exercise trials to adequately address post-exertional malaise. It addresses the overlaps between ME/CFS and Long COVID and provides some back-history of the PACE trial debacle. It’s great to see New Scientist taking a deeper look at some of the usual claims.