By David Tuller, DrPH
I have always made it clear that I pay attention when smart patients assess bad research. That’s how I stumbled into this whole mess in the first place–by reading what patients were writing about the PACE trial. (In that case, I at first dismissed the concerns when I read about how participants could get worse and still be counted as recovered. I figured patients must have it wrong. I knew no credible journal could have published such a study–much less failed to retract it when the problem had been pointed out.)
One of the smart patients whose analyses I pay attention to is Michiel Tack. He was part of a lively discussion on the Science For ME forum of the so-called “feasibility study” of cognitive behavior therapy plus music therapy as a treatment for chronic fatigue after acute Epstein-Barr virus infection in adolescents, which I have criticized here, here and here. In reality, the journal, BMJ Paediatrics Open, published a fully powered randomized trial that failed to meet expectations as if it were always intended to be a feasibility study for a fully randomized trial.
Michiel submitted a cogent and meticulously argued comment to the journal but has not received a response. He has now posted the letter on the PubPeer site here. I am reposting it on Virology Blog with his permission.
**********
Malik et al. conducted a randomized trial of cognitive-behavioral therapy (CBT) combined with music therapy for adolescents with chronic fatigue (CF) following Epstein-Barr virus infection. (1) Unfortunately, there are several problems with how the findings of this trial are reported.
First, it appears that the study was rephrased as a feasibility trial when the intervention failed to provide the expected effect sizes. The trial was only powered to detect large effects and both the protocol (2) and statistical analysis (3) plan suggest that the authors were expecting to find large improvements in the intervention group. In their power calculation for the primary outcome (mean number of steps per day) they wrote: “In the present study, the power to detect an increment of 2000 steps/day is at least 80 % (α=0.05). This effect size is rather large (0.8 times the standard deviation); however, as CBT alone is documented to have small to moderate effect size in CFS/ME, only a substantial effect size is of direct clinical interest. Also, the FITNET study suggests that larger treatment effects might be assumed in adolescent CFS/ME patients as compared to adults (Nijhof 2012).” (2) The protocol (2) and trial registration (4) include more than 20 outcome measures suggesting the study aimed to test the efficacy rather than the feasibility of the intervention.
Second, Malik et al. conclude that “combined CBT and music therapy is feasible and acceptable in adolescent postinfectious CF.” (1) The authors refer to high compliance and a lack of statistically significant differences in reported harms between the intervention and control group. The latter, however, could also be explained by a lack of power as the trial had only 43 participants and could only detect large differences between the groups. There were also some indications that the intervention might not be acceptable to patients. More than half of the eligible adolescents refused participation in the trial. In the protocol (2) and statistical analysis plan (3), the authors assumed that only 5 percent would decline participation. There was also a higher drop-out rate in the intervention (29%) compared to the control group (5%).
Third, according to Malik et al. “the study indicates that the mental training programme may improve symptoms and lead to higher recovery rate over time.” (1) This statement is not supported by the data. The primary outcome measure was mean steps per day assessed with an accelerometer 12 weeks post-randomization. For this outcome, the intervention group showed lower activity levels than the control group, a difference that reached statistical significance in the per-protocol but not the Intention-to-treat analysis. Unfortunately, the authors do not discuss this rather surprising finding. Patients in the control group only received “care as usual.” The paper explains that “’care as usual’ implies that the relevant individuals would not receive any healthcare for their CF condition in the follow-up period apart from the follow-up visits in the present study.” (1) So it seems that for the primary outcome, the intervention did worse than doing nothing. The time and energy spent on CBT and music therapy could have prevented patients from resuming their normal activities. This explanation is in line with a high drop-out rate in the intervention group and more than half of eligible patients declining to participate in the study.
Fourth, the authors highlight trends toward statistical significance even though more than 20 secondary outcomes measures were used in the study (each taken at two time frames) while no corrections for multiple comparisons were performed. It, therefore, seems inappropriate to highlight that the intervention “may improve […] symptoms of postexertional malaise, justifying a full-scale trial.” Postexertional malaise was assessed with a single question and seems to be added post hoc as it is not listed as an outcome in the protocol (1), statistical analysis plan (2), or trial registration (3). The maximum difference for postexertional malaise between the intervention and control group was only 0.5 points, a third of the standard deviation for the intervention group at baseline. In the intention-to-treat analysis, this difference decreases to 0.2 points at follow-up. It is unclear why these results would justify a full-scale trial given that the intervention group did worse on the primary outcome than the control group.
Fifth, the abstract also highlights a trend towards a higher recovery rate in the intervention group. The authors defined recovery as a score lower than 4 points on the Chalder Fatigue Scale using a dichotomous scoring method (range 0-11 points). A score of 4 or higher on the Chalder Fatigue Scale, however, was already used as an inclusion criterium. This means that participants could be classified as recovered as a result of reporting an improvement of just 1 point on the Chalder Fatigue Scale. It should also be noted that the Chalder Fatigue Scale does not assess the intensity or impact of fatigue. Instead, it assesses whether participants experience fatigue-related symptoms such as having ”problems starting things” or finding it “more difficult to find the right word” more than usual. Consequently, it seems inappropriate to use the term “recovery rate” for the percentage of participants who score lower than the threshold of 4 points on the Chalder Fatigue Scale. Post-treatment there was no difference in the percentage of patients meeting this 4-point threshold between the intervention and control group. The intention-to-treat analysis of the Chalder Fatigue Scale ordinal scoring (range 0-33) was also reported and this showed little difference between the two groups. A plausible explanation for what the authors describe as “a trend towards higher recovery rate in the intervention group” is the high drop-out rate. At follow-up, only 13 patients were in the intervention group. The analysis highlighted by the authors does not take into account the 8 persons who were in the intervention group but were lost to follow-up. There are little reasons to suggest that more patients in the intervention group recovered than in the control group. It is unfortunate that the authors have used this term in their manuscript.
In conclusion, CBT combined with music therapy was associated with a high drop-out rate and lower activity levels than participants who received no intervention. Contrary to what Malik et al. conclude these results question whether a full-scale clinical trial is justified.
References
- Malik S, Asprusten TT, Pedersen M, Mangersnes J, Trondalen G, Roy B van, et al. Cognitive–behavioural therapy combined with music therapy for chronic fatigue following Epstein-Barr virus infection in adolescents: a feasibility study. BMJ Paediatr Open. 2020 Apr 1;4(1):e000620.
- Akershus University Hospital. Research Protocol – processing. Mental training for chronic fatigue syndrome (CFS/ME) following EBV infection in adolescents: a randomised controlled trial. Available from: https://www.ahus.no/seksjon/forskning/Documents/Forskningsgrupper/Barne- og ungdomsklinikken/Paedia/Forskningsprotokoll – behandling.pdf
- Akershus University Hospital. Statistical analysis plan – CEBA part 2. Available from: https://www.ahus.no/seksjon/forskning/Documents/Forskningsgrupper/Barne- og ungdomsklinikken/Paedia/Statistisk analyseplan del 2.pdf
- ClinicalTrials.gov Identifier: NCT02499302. Available from: https://clinicaltrials.gov/ct2/show/NCT02499302
Comments are closed.