By David Tuller, DrPH
UPDATE: I sent the following correction to Dr Choonara shortly after sending the letter of concern.
Dear Dr Choonara:
I wanted to make a slight correction in point #3 below. The first sentence should have read: “Why was the outcome of recovery not mentioned in the trial registration and statistical analysis plan yet still highlighted in the paper?” As I previously reported on Virology Blog, the outcome of recovery was mentioned in the protocol but not in the other two trial documents. I apologize for the error.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Berkeley, California, USA
**********
I sent the following letter today to the editor-in-chief of BMJ Paediatrics Open–Imti Choonara, Emeritus Professor in Child Health at University of Nottingham. I cc’d Fiona Godlee, editorial director of BMJ, to alert her to the distressing news that yet another one of her high-profile journals has published more research from the CBT ideological brigades that violates core scientific principles.
**********
Dear Professor Choonara:
BMJ Paediatrics Open recently published a study called “Cognitive-behavioural therapy combined with music therapy for chronic fatigue following Epstein-Barr virus infection in adolescents: A feasibility study.” We are writing to express our concerns about what appear to be significant methodological problems with this paper.
The science site Virology Blog has published three posts critical of the paper–here, here and here. A cogent and meticulously argued comment that has been posted under the article itself raises many related points. (Virology Blog is a science site hosted by Columbia University microbiologist Vincent Racaniello, one of the signers of this letter. The relevant posts were written by UC Berkeley journalist and public health expert David Tuller, another signer.)
The study’s primary outcome was an objective one, average steps per day, measured with an accelerometer worn for a week. To the surprise of the investigators, the average number of steps in both study arms dropped, with the intervention group performing even worse than the treatment-as-usual comparison group on this measure. Secondary outcomes mentioned in the paper, including post-exertional malaise (PEM) and fatigue, were subjective. The investigators concluded that a full trial might be justified.
These are among the many concerns that we believe need to be addressed:
1) The trial protocol, registration and statistical analysis plan all describe the research as a fully powered trial and not as a feasibility study. But recruitment for the trial apparently proved difficult, the intervention group experienced high attrition, and the results were disappointing. Why did BMJ Paediatrics Open publish a fully powered randomized trial that failed to meet expectations as a feasibility study designed to explore the possibility of a fully powered randomized trial?
2) Why was PEM highlighted as an outcome in the published paper when it was not mentioned in the trial protocol, registration, or statistical analysis plan? Did anyone at BMJ Paediatrics Open review the trial protocol, registration and statistical analysis plan to check whether the investigators had engaged in undisclosed outcome-switching?
3) Why was the outcome of recovery not mentioned in the trial protocol and statistical analysis plan yet still highlighted in the paper? Why did the main text focus on a per-protocol analysis of these findings rather than an intention-to-treat analysis, which would have yielded a lower recovery rate for the intervention arm by taking into account the high attrition rate?
4) Why did the investigators construct a definition of recovery that excluded the results of the study’s own objective primary outcome and relied solely on a subjective secondary outcome? If participants were worse off in terms of average steps per day than before the intervention, how could they reasonably be said to have achieved recovery, at least as the term is commonly understood?
5) In making a case that a further trial might be justified, why did the conclusions in the abstract and full paper omit mention of the poor results for the primary objective outcome and rely instead on results for subjective secondary outcomes? Why did the conclusions not mention the difficulty the investigators faced in recruitment and the high attrition rate in the intervention group?
An examination of the peer review history also raises questions about the rigor of BMJ Paediatrics Open‘s editorial stewardship. Although one reviewer stated flatly that he had not read “beyond the abstract,” the acknowledgement apparently did not present an obstacle to publication in BMJ Paediatrics Open. Does the journal consider a peer review of an abstract to be a robust substitute for a peer review of a full paper?
Moreover, the other reviewer, who presumably read the whole paper–expressed confusion over whether the investigators were describing a fully powered randomized trial or a feasibility study. The reviewer asked them to clarify the original aim of the research.
Instead of clarifying the original aim, the investigators rewrote the history of the project by presenting it as if it had been a feasibility study from the start. In light of the reviewer’s confusion, did BMJ Paediatrics Open vet the revision against the trial documents to ensure that the investigators’ new description of the study was a transparent and accurate account of what they had intended?
It cannot reasonably be argued that this study underwent a peer review process consistent with BMJ’s professed standards. Under the circumstances, BMJ Paediatrics Open should temporarily withdraw the paper and assign it out again for another review. Any new peer reviewer must promise to read beyond the abstract, and the review must include scrutiny of the trial documents as well as the paper. Should the new review and BMJ Paediatrics Open’s own examination of the matter confirm major discrepancies between what the investigators promised and what they published, the study should be retracted.
For full transparency, this letter will be posted on Virology Blog. Given the serious methodological violations involved and the apparent gaps in BMJ Paediatrics Open‘s editorial oversight practices, we have cc’d Dr Fiona Godlee, editorial director of BMJ.
We hope you promptly address the issues with this problematic paper.
Best–
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, England, UK
Mady Hornig, MA, MD
Associate Professor of Epidemiology
Columbia University Mailman School of Public Health
New York, New York, USA
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York, USA
John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Berkeley, California, USA
Comments are closed.