By David Tuller, DrPH
Earlier today, Professor Racaniello e-mailed the following letter to Nick Brown, the editor-in-chief of Archives of Disease in Childhood, one of the journals from the BMJ Group. Archives recently published a study of the Lightning Process in kids with ME/CFS.
**********
Dear Dr. Brown:
In September, Archives of Disease in Childhood published a study called Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial [1]. Like many studies of this illness, it was an open-label trial with self-reported outcomes, a study design known to be highly vulnerable to bias.
The researchers have received criticism for their initial decision to conduct a pediatric trial of the Lightning Process–a commercial, pseudo-scientific intervention that combines elements of life-coaching, osteopathy, and neurolinguistics programming. The intervention encourages participants to report that it has made them better, another source of potential bias. Beyond the decision to study the Lightning Process, the study suffered from major cross-contamination between treatment arms and significant loss-to-follow-up, among other problems.
In addition to these issues, we are also concerned about a methodological anomaly reported on Virology Blog by David Tuller from the University of California, Berkeley: An earlier feasibility trial was extended into the full trial even as the researchers swapped primary and secondary outcomes following a review of the feasibility trial results [2]. The researchers received ethical approval for these study design changes but failed to mention them in the Archives paper, although the changes enabled the findings to be presented in a much more positive light than would otherwise have been the case.
These changes are only apparent from a review of other trial-related documents, such as the registration [3], the feasibility trial protocol [4] and the full-trial protocol [5]. Our specific concerns include the following:
*The researchers registered the study as a prospective trial. But more than half the sample were apparently participants recruited for the feasibility trial [6], starting almost two years before the registration date. The Archives paper did not mention this fact or explain the discrepancy in dates.
*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After reviewing results from the feasibility trial, the researchers swapped these outcome measures for the full trial, school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. The Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had provided data as participants in the feasibility trial.
*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant mention in the Archives paper. Thus the outcome-swapping allowed the researchers to report positive results for their primary outcome. Had they not switched the outcomes after more than half the sample had provided data, they would have had to report null primary outcome results
*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a complement to self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. The researchers offered no explanation for the absence of these official school attendance records. Interestingly, it was stated in a recent news account that the official school attendance records corroborated the self-reported school attendance results, suggesting that these data were in fact collected.
*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results for school attendance at six months, the original primary outcome, were largely ignored. The outcome-swapping that occurred after more than half the sample had provided data, and that the Archives paper did not mention, ensured that the reported findings received more positive media coverage than they otherwise would have.
Transparency is essential in scientific research. In this case, the Archives paper failed to disclose salient and relevant information about the composition of the study sample, the outcome-swapping, and the official school attendance records. These missing details, if provided, would help others to more properly assess the validity and reliability of the reported findings.
The Archives paper must be corrected to acknowledge that the sample included the feasibility trial participants and that the review of their data informed the decision to switch the outcome measures. The paper should also present separate results for participants in the feasibility trial and those recruited afterwards. And the corrected version must clarify the status of the official school attendance records; if these data were collected, as the recent news account suggested, they should also be reported.
We urge Archives of Disease in Childhood to take these concerns seriously and address them promptly.
Sincerely,
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, NY, USA
Lisa F. Barcellos, MPH, PhD
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California, Berkeley
Berkeley, CA, USA
Robin Callender Smith, PhD
Professor of Media Law
Centre for Commercial Law Studies
Queen Mary University of London
Barrister and Information Rights Judge
London, UK
Lily Chu, MD, MSHS
Independent Consultant
Burlingame, CA, USA
Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, CA, USA
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, UK
Valerie Eliot Smith
Barrister and Visiting Scholar
Centre for Commercial Law Studies
Queen Mary University of London
London, UK
Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, VA, USA
Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, IL, USA
Michael W. Kahn, MD
Assistant Professor of Psychiatry
Harvard Medical School
Boston, MA, USA
Nancy Klimas, MD
Director, Institute for Neuro-Immune Medicine
Nova Southeastern University
Director, Miami VA Medical Center GWI and CFS/ME Program
Miami, FL, USA
Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, NY, USA
Steven Lubet
Williams Memorial Professor of Law
Northwestern University Pritzker School of Law
Chicago, IL, USA
Marlon Maus, MD, DrPH, FACS
Director, DrPH Program
School of Public Health
University of California, Berkeley
Berkeley, CA, USA
Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, VA, USA
Zaher Nahle, PhD, MPA
Vice President for Research and Scientific Programs
Solve ME/CFS Initiative
Los Angeles, CA, USA
Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, CA, USA
John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, CA, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
University of California, Berkeley
Berkeley, CA, USA
William Weir, FRCP
Infectious Disease Consultant
London, UK
Carolyn Wilshire, PhD
Senior Lecturer
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
**********
[1] Crawley EM, Gaunt DM, Garfield K, et al. 2018. Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 103(2):155-164
[2] Tuller D. 2017. Trial by error: the SMILE trial’s undisclosed outcome-swapping. Virology Blog; Dec 13. Accessed at: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/
[3] ISRCTN Registry: ISRCTN8145620. July 31, 2012. Specialist Medical Intervention & Lightning Evaluation: Comparing specialist medical care with specialist medical care plus the Lightning Process for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME). Accessed at: http://www.isrctn.com/ISRCTN81456207
[4] Crawley EM, et al. July 2010. SMILE Protocol v6: Assessing the feasibility and acceptability of comparing the Lightning Process with specialist medical care for Chronic Fatigue Syndrome or Myalgic Encephalopathy (CFS/ME)–pilot Randomised Controlled Trial. Accessed at: http://www.bristol.ac.uk/media-library/sites/ccah/migrated/documents/smprotv6final.pdf
[5] Crawley EM, Mills N, Hollingworth W, et al. 2013. Comparing specialist medical care with specialist medical care plus the Lightning Process® for chronic fatigue syndrome or myalgic encephalomyelitis (CFS/ME): study protocol for a randomised controlled trial (SMILE Trial). Trials; 14:444
[6] Crawley EM, Mills N, Beast L, et al. 2013 The feasibility and acceptability of conducting a trial of specialist medical care and the Lightning Process in children with chronic fatigue syndrome:: feasibility randomized controlled trial (SMILE study). Trials; 14:415.
Comments are closed.