By David Tuller, DrPH
Alan Montgomery is a professor of medical statistics and clinical trials at the University of Nottingham’s School of Medicine. He is also the senior author of the Lightning Process study, which was published in 2017 in Archives of Disease in Childhood, a BMJ journal. I wrote him a letter in January of this year, alerting him to my concerns about the study, but did not hear back. Because a recent review cited this problematic trial in highlighting the Lightning Process as an “effective” treatment for children, I decided it was time to send Professor Montgomery a second letter reminding him about my first.
I sent the following earlier today.
**********
Dear Professor Montgomery–
I am a Senior Fellow in Public Health and Journalism at the Center for Global Public Health, part of the School of Public Health at the University of California, Berkeley. Last January, I wrote to you about some problems with a paper on which you were the senior author: Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.” Archives of Disease in Childhood, a BMJ journal, published this paper in September, 2017. Your co-authors were all from the University of Bristol.
I did not receive a response to that January letter. However, I have stepped up my efforts to bring attention to this issue since April, when Current Opinion in Pediatrics published a review called “Child and adolescent chronic fatigue syndrome/myalgic encephalomyelitis: where are we now?” That review touted the Lightning Process as “effective” for treating children, based solely on your study. Since the review has the potential to impact clinical guidelines and policy decisions related to a vulnerable population, it is essential that those involved in disseminating this claim acknowledge the flaws in the study on which it is based. Protecting children from possible harms related to questionable interventions should be an overriding priority.
I first documented some ethical and methodological problems with the Lightning Process paper in a post on Virology Blog in December, 2017. In January, 2018, Professor Vincent Racaniello, a Columbia University microbiologist and the host of Virology Blog, sent a letter of concern to Dr Nick Brown, Archives’ editor-in-chief, signed by 21 scientists and other experts. In June, 2018, Archives posted an “editor’s note” that essentially confirmed irregularities in the trial registration and the reporting of results. This editor’s note also indicated that you and your co-authors had submitted “clarifications” and that the matter was under “editorial consideration.” On June 19th, that period of “editorial consideration” will have lasted a full year.
As I outlined in my January letter to you, my specific concerns about the study and paper include the following:
*The study was registered as a prospective trial. But 56 out of 100 participants were recruited as part of the feasibility trial, starting almost two years before the registration date. For unexplained reasons, the Archives paper did not mention this fact. Like all major publishers of medical journals, BMJ has a longstanding policy of not considering any clinical trials in which participants were recruited before registration. In considering and then publishing the Lightning Process paper, Archives of Disease in Childhood violated its own stringent guidelines.
*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After the collection of data for the feasibility trial, these outcome measures were swapped for the full trial, school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. As with the trial registration issue, the Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had been recruited as participants in the feasibility trial.
*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant attention in the Archives paper. Thus, the outcome-swapping biased the findings and improved the apparent success of the intervention. Had the measures not been switched, the paper would have had to report null results for the primary outcome.
*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a way of vetting the accuracy of self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. No explanation for the absence of these official school attendance records was provided, leaving two possibilities: Either they were not obtained, or they were obtained and failed to confirm the benefits in self-reported school attendance at 12 months.
*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results reported for school attendance at six months, the original primary outcome, were largely ignored. The undisclosed outcome-swapping thus ensured that the reported findings received more positive media coverage than they otherwise would have.
Professor Montgomery, as I wrote you last January–this situation has dragged on for too long. The facts are not in dispute, and children’s health and well-being are at stake. I urge you, as senior author of the Lightning Process study, to take appropriate action. Under the circumstances, that means retracting the paper while providing a fully transparent account of what went wrong and how it went wrong.
In an effort to expedite action on this matter, I am cc-ing several people: Dr Fiona Godlee, editorial director of BMJ; Dr Nick Brown, editor-in-chief of Archives of Disease in Childhood; Sue Paterson, director of legal services at Bristol University; Professor Racaniello; Professor Chris Ponting, current vice-chair of the CFS/ME Research Collaborative; Dr Terry Segal, senior author of the recent review in Current Opinion in Pediatrics; four doctors involved in the efforts to develop new ME/CFS guidelines under the auspices of the National Institute for Health and Care Excellence, along with two patient representatives to the NICE committee; and two members of Parliament who have expressed concern about the quality of ME/CFS research, along with a parliamentary aide.
Thank you for your attention to this matter.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.