By David Tuller, DrPH
I’ve recently spent some time lambasting a Long Covid study in The BMJ that claimed a rehab program addressing both physical and mental health was “clinically effective”—even though the primary outcome results fell below the recommended level for what would be considered “minimal clinically important difference” on the measure in question. Now another high impact journal has published another Long Covid study that also engages in unacceptable methodological shenanigans.
Here’s the title of the new study, which examines the impact of physical activity (PA) in young women with what is referred to as post-covid condition (PCC): “Higher level of physical activity reduces mental and neurological symptoms during and two years after COVID-19 infection in young women.” And here’s the summation of the findings from the abstract: “In conclusion, a higher level of regular PA can reduce…the number of mental and neurological symptoms in PCC underlying the importance of regular PA, even in this and likely other viral disease conditions.”
The authors of the paper are from Semmelweiss University in Budapest; the senior author is also affiliated with the Hungarian University of Sports Science in Budapest and New York Medical College in Valhalla, New York. The paper was published this week by Scientific Reports, a prestigious journal under the Nature publishing umbrella. Not surprisingly, the journal claims to have an “extensive network of expert peer reviewers” and promises that “our editorial team provides rigorous, objective and constructive peer review.”
These claims are bullshit, at least as far as this paper is concerned. This is clear not just from the title but from the conclusion of the study as well. The use of the verb “reduce” in both indicates a causal link. In other words, the title and conclusion are declaring that the greater level of physical activity led to the reduction of symptoms in women. That would be great, if it were a valid statement. Unfortunately for the study authors, it is a statement that cannot legitimately be made based on the study itself—a cross-sectional survey.
Cross-sectional surveys can only find associations. They are not designed to illuminate causal relationships. The reason is obvious. If you’re measuring two things at the same time, it can be impossible to know the order of events. Which caused what? Who knows?
Here’s an easy example. A cross-sectional survey of gay men might find that those with HIV were less likely to use condoms than gay men without HIV. (Let’s postulate that this survey was done years ago, before the advent of pre-exposure prophylaxis for HIV muddied the waters a bit.) A possible assumption—similar to the one made by the authors of the Hungarian study—might be that the failure to use condoms led to the HIV infections.
But to assert such a relationship as a fact would be completely unjustified. I mean, it’s possible, of course, that the chain of events occurred in that order for some of the respondents. But the association between lack of condom use and HIV infection can also be explained the other way around—those who know they have HIV are less likely to use condoms when they have sex because they’re not worried about contracting HIV. In the case of such a survey, the causal relationships are most likely running in both directions; both explanations are viable given the data from the survey. But it is impossible to know which came first.
Similar interpretive confusion about the meaning of associations plagues many of the biopsychosocial studies of the illness or cluster of illnesses referred to as ME/CFS. Studies often show that those who attribute their condition to physical disease are more likely to have worse symptoms. This has been interpreted by clueless investigators—King’s College London’s mathematically challenged Trudie Chalder, a professor of cognitive behavioural therapy, comes to mind as a prime example—as evidence that the belief of physical disease is itself leading to negative outcomes. But the alternate interpretation—that those who are physically sicker are correctly identifying the source of their ill more and are more likely to be doing much worse—is just as reasonable an assumption. The truth is that the data do not allow anyone to distinguish between the two explanations.
The study in Scientific Reports analyzed the responses of 802 women from 18 to 34; of the more than 50% who had experienced an acute bout of Covid-19, the average time since infection was almost two years. The investigators divided the participants into categories of low, moderate and high physical activity based on how many minutes a week they reported engaging in physical activity. At the same time, they assessed dozens of symptoms using the WHO’s form for what the agency calles “post COVID condition.” Around a quarter of the women who had been infected reported ongoing persistent or intermittent symptoms that were “primarily neurological and mental,” according to the study.
In outlining the goal of the reesarch, the investigators wrote the following: “This study aimed to examine the hypothesis that regular PA reduces the number of symptoms during and after COVID-19 infection in young women.” It should have been obvious to any smart person reading the draft that the cross-sectional study design did not allow for any such examination. It is disturbing that this paper passed peer review in this form. The title and the entire thrust of the paper require corrections.
“A cross-sectional survey of gay men might find that those with HIV were ” now more ” likely to use condoms than gay men without HIV.” This could be misinterpreted as condom use increased the likelihood of HIV which is misleading if the usage behaviour was vastly different before they knew they had HIV.
It’s more likely that being HIV positive makes you more safety conscious & wanting to protect others but these statements are still unsupported by the hypothetical study. The assumptions & claims have to be better tested & evidence of causation not just correlation.
Hey there Tygrus–I would agree in regards to HIV+ guys having sex with HIV- guys, in terms of many taking precautions. But there was a point where, for understandable reasons, many HIV+ guys did not want to have sex with HIV- guys because of the anxiety and because it required the use of condoms. HIV+ guys having sex with each other were often not using condoms long before PrEP came into being.
Surely doctors either have to be able to trust what they read in medical journals or it needs to be impressed on them that they can’t rely on any of it to be accurate/truthful? I’d say that we’ve reached a point in medical journalism where the latter’s the only safe option until journals demonstrate that they take their responsibilities towards patient welfare seriously.