By David Tuller, DrPH
I have sent the following letter to Dr Fiona Godlee, editorial director of BMJ, about Archives of Disease in Childhood’s egregious decision to re-publish Bristol University’s Lightning Process trial with the original findings intact. Given the Bristol team’s flagrant methodological violations, the journal should have retracted the paper.
Failing that, Archives should have required the investigators to re-publish the paper with the initial primary outcome–school attendance at six months, which generated null results–as the final primary outcome. Dr Godlee was well aware of the issues involved, so she presumably endorsed the journal’s actions. The documented facts suggest that BMJ’s ethical compass is broken and that editors are more concerned about possible reputational damage than about the integrity of the medical literature or the health and well-being of children. That is presumably why dozens of scientists, clinicians and other experts from top academic institutions were so eager to sign this letter.
**********
Dear Dr Godlee—
We are writing about the correction appended last month [1] to the pediatric study of the Lightning Process conducted by investigators from the University of Bristol and published by Archives of Disease in Childhood in September, 2017. The study appeared under the following title: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.€ [2]
As Virology Blog documented shortly after publication [3], the investigators recruited more than half the sample before trial registration as part of a feasibility study, swapped primary and secondary outcomes after gathering data from these early participants, and did not disclose these salient details in the paper. With the recent correction and an accompanying editor€™s note, Archives of Disease in Childhood has acknowledged the validity of Virology Blog€™s criticisms. Yet the journal has allowed the investigators to reassert their main conclusion–that the Lightning Process is €œeffective€ in treating pediatric illness. This decision is untenable.
The Lightning Process is a commercial program combining life-coaching, neuro-linguistic programming and positive affirmations. Participants are taught that they can overcome illness by controlling and changing their thought patterns. Lightning Process practitioners have declared that they can successfully treat multiple sclerosis, eating disorders and other serious conditions. Government regulators have admonished practitioners for making misleading claims.
Beyond concerns about the Lightning Process itself, the Bristol study was an open-label trial with self-reported outcomes, a combination of design features that can lead to significant bias. Furthermore, in violating core methodological and ethical principles of scientific inquiry, the investigators also breached BMJ€™s own strict policy against publishing any trials in which participants have been recruited before registration. Since all major medical journals adopted this policy in 2005, experienced investigators, including members of the Bristol team, should have been well aware of it.
BMJ has been a leader in the movement to require prospective registration as a condition for trials to be considered for publication. In 2013, in testimony before the House of Common€™s Science and Technology Committee, you touted BMJ€™s success in implementing a zero-tolerance approach to such trials. €œAs far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,€ you stated, a claim you could not accurately repeat today. [4] A 2015 editorial in The BMJ called the prospective registration policy €œthe single most valuable tool we have to ensure unbiased reporting of research studies.€ [5]
Given BMJ€™s longstanding position that prospective registration is essential for safeguarding the integrity of the medical literature, it is unclear why you are now willing to exempt the Lightning Process study from this requirement. The editor€™s note posted with the correction declared that lack of prospective registration was not grounds for retraction. [6] No explanation or justification was provided for this determination, which appears to be incompatible with the zero-tolerance approach you advocated in your parliamentary testimony.
As you certainly know, a key goal of prospective registration is to prevent selective outcome reporting. Yet selective outcome reporting is exactly what occurred in the Lightning Process study when the investigators swapped primary and secondary outcomes mid-way through. In the published paper, the investigators reported positive results for the revised primary outcome of self-reported physical function at six months, which had initially been designated a secondary outcome. In contrast, they reported null results for the original primary outcome of school attendance at six months, which by the end had been demoted to a secondary outcome.
The swap thus ensured that the investigators were able to report positive rather than null results for their official primary outcome. Not surprisingly, this change had a major impact on how the information was presented in Archives of Disease in Childhood and interpreted by news organizations. The positive findings for the primary outcome that was selected after more than half the participants had provided data were prominently highlighted in the published paper. These findings received widespread media attention [7] and have recently been cited in a systematic review as evidence that the intervention is “effective.” [8] The null findings for the original primary outcome were ignored.
The editor€™s note explains that the journal examined the outcome-swapping issue by €œseeking assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.€ This statement is perplexing. Individuals subject to potential bias are not generally considered impartial and authoritative arbiters of whether this potential bias has influenced their decision-making. Study design is supposed to seek to minimize bias precisely because humans tend to be blind to their own biases. Reputable science in high-impact journals should not have to rely on investigators€™ €œassurance€ that they have resisted the natural temptation to let their perceived interests guide their selection of primary outcomes.
Moreover, in relation to any assurances provided by the Bristol investigators, it is worth remembering that they withheld important information about trial registration and outcome-swapping from their public version of events. Archives of Disease in Childhood would not have published the paper in the first place except for two major failures: 1) The investigators€™ failure to provide an accurate account of how they conducted the trial; and 2) The journal€™s failure to detect disqualifying flaws, despite its obligation to subject the paper to rigorous peer review and editorial oversight.
Archives of Disease in Childhood has now re-published positive primary results generated by outcome-swapping in a study that did not meet a strict publication requirement specifically intended to prevent outcome-swapping and other kinds of selective outcome reporting. In doing so, the journal has rewarded the Bristol investigators for their lack of candor and their methodological missteps and has contradicted BMJ€™s and your own past statements about the critical importance of prospective registration.
In this case, BMJ€™s actions are scientifically and ethically indefensible. They are also potentially harmful to the health and well-being of children. Thank you for your prompt attention to this troubling matter.
Best–
Dharam V. Ablashi, DVM, MS, Dip Bact
Scientific Director
HHV-6 Foundation
Santa Barbara, California, USA
Former Senior Investigator
National Cancer Institute
National Institutes of Health
Bethesda, Maryland, USA
Michael Allen, PhD
Clinical Psychologist (retired)
San Francisco, California, USA
James N. Baraniuk, MD
Professor of Medicine
Georgetown University
Washington, DC, USA
Lisa F. Barcellos, MPH, PhD
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California, Berkeley
Berkeley, California, USA
Lucinda Bateman, MD
Medical Director
Bateman Horne Center
Salt Lake City, Utah, USA
Alison C. Bested, MD, FRCPC
Clinical Director
Institute for Neuro-Immune Medicine
Associate Professor
Nova Southeastern University
Fort Lauderdale, Florida, USA
Charlotte Blease, PhD
Fulbright and Marie Curie Research Fellow
General Medicine and Primary Care
Beth Israel Deaconess Medical Center
Harvard Medical School
Boston, Massachusetts, USA
Molly Brown, PhD
Assistant Professor
Department of Psychology
DePaul University
Chicago, Illinois, USA
Robin Callender Smith, PhD
Professor of Media Law
Centre for Commercial Law Studies
Queen Mary University of London
Barrister and Information Rights Judge
London, England, UK
Janet L Dafoe, PhD
Child Psychologist in Private Practice
Palo Alto, California, USA
Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, California, USA
Simon Duffy, PhD, FRSA
Director
Centre for Welfare Reform
Sheffield, England, UK
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, England, UK
Valerie Eliot Smith
Barrister and Visiting Scholar
Centre for Commercial Law Studies
Queen Mary University of London
London, England, UK
Margaret C Fernald, PhD
Clinical Child Psychologist
Clinical Associate of Psychology
University of Maine
Orono, Maine, USA
Kenneth J. Friedman, PhD
Associate Professor of Physiology and Pharmacology (retired)
New Jersey Medical School
University of Medicine and Dentistry of New Jersey
Newark, New Jersey, USA
Robert F. Garry, PhD
Professor of Microbiology and Immunology
Tulane University School of Medicine
New Orleans, Louisiana, USA
Claudia Gillberg, PhD
Fellow, Centre for Welfare Reform
Sheffield, England, UK
Senior Research Associate
National Centre for Lifelong Learning
Jonkoping University
Jonkoping, Sweden
Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, Virginia, USA
Alan Gurwitt, MD
Psychiatrist in Private Practice (retired)
Associate Clinical Professor
Yale Child Study Center (retired)
New Haven, Connecticut, USA
Associate Clinical Professor
University of Connecticut Dept of Psychiatry (retired)
Storrs, Connecticut, USA
Lecturer, Harvard Medical School (retired)
Boston, Massachusetts, USA
Geoffrey Hallmann, LLB, DipLegPrac, DipFinPrac
Specialist in Disability and Compensation Law (retired)
Lismore, New South Wales, Australia
Brian M. Hughes, PhD, FPsSI
Professor of Psychology
National University of Ireland, Galway
Galway, Ireland
Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, Illinois, USA
Maureen Hanson, PhD
Liberty Hyde Bailey Professor
Department of Molecular Biology and Genetics
Cornell University
Ithaca, New York, USA
Keith Kahn-Harris, PhD
Associate Lecturer in Sociology
Birkbeck, University of London
London, England, UK
Jon D. Kaiser, M.D.
Clinical Faculty
Department of Medicine
University of California, San Francisco
San Francisco, California, USA
Nancy Klimas MD
Director, Institute for Neuro-Immune Medicine
Nova Southeastern University
Director, Miami VA Medical Center GWI and CFS/ME Program
Miami, Florida, USA
Eliana M. Lacerda, MD, MSc, PhD
Clinical Assistant Professor
International Centre for Evidence in Disability
Faculty of Infectious and Tropical Diseases
London School of Hygiene & Tropical Medicine
London, England, UK
Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, New York, USA
Rogier Louwen, PhD
Assistant Professor
Department of Medical Microbiology and Infectious Diseases
Erasmus University Medical Center
Rotterdam, The Netherlands
Steven Lubet, JD
Williams Memorial Professor of Law
Northwestern University Pritzker School of Law
Chicago, Illinois, USA
Marlon Maus, MD, DrPH, FACS
Director, DrPH Program
School of Public Health
University of California, Berkeley
Berkeley, California, USA
Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, Virginia, USA
Heidi Nicholl, PhD
Chief Executive Officer
Emerge Australia
Melbourne, Victoria, Australia
Steve Olson, MD
Family Physician
Large Medical Group Executive and Regional Physician Director
Oakland, California, USA
Elisa Oltra, PhD
Professor of Molecular and Cellular Biology
Catholic University of Valencia School of Medicine
Valencia, Spain
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York, USA
Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley
Berkeley, California, USA
Ola Didrik Saugstad, MD, PhD
Professor Emeritus
Division of Paediatric and Adolescent Medicine
University of Oslo
Oslo, Norway
Adjunct Professor of Pediatrics
Feinberg School of Medicine
Northwestern University
Chicago, Illinois, USA
David Scales MPhil, MD, PhD
Assistant Professor of Medicine
Division of Hospital Medicine
Weill Cornell Medicine
New York, New York, USA
Michael Scott, PhD
Clinician and Researcher
Psychological Therapies Unit
Liverpool, England, UK
Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, California, USA
Eleanor Stein, MD, FRCP(C)
Psychiatrist in Private Practice
Assistant Clinical Professor
University of Calgary
Calgary, Alberta, Canada
Leonie Sugarman, PhD
Emeritus Associate Professor of Applied Psychology
University of Cumbria
Carlisle, England, UK
John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California, USA
Ronald G. Tompkins, MD, ScD
Summer M Redstone Professor of Surgery
Harvard Medical School
Boston, Massachusetts, USA
Samuel Tucker, MD
Assistant Clinical Professor of Psychiatry (retired)
University of California, San Francisco
San Francisco, California, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
University of California, Berkeley
Berkeley, California, USA
Rosamund Vallings, MNZM, MBBS
General Practitioner
Auckland, New Zealand
Linda van Campen, MD
Cardiologist
Stichting Cardiozorg
Hoofddorp, The Netherlands
Mark Vink, MD
Family Physician
Soerabaja Research Center
Amsterdam, The Netherlands
Frans Visser, MD
Cardiologist
Stichting Cardiozorg
Hoofddorp, The Netherlands
Tony Ward, MA (Hons), PhD, DipClinPsyc
Registered Clinical Psychologist
Professor of Clinical Psychology
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
Adjunct Professor, School of Psychology
University of Birmingham
Birmingham, England, UK
Adjunct Professor, School of Psychology
University of Kent
Canterbury, England, UK
John Whiting, MD
Specialist Physician in Private Practice
Brisbane, Queensland, Australia
Carolyn Wilshire, PhD
Senior Lecturer
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
**********
[1] Crawley E, Gaunt D, Garfield K, et al. Notice of correction and clarification: Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 2019. Accessed at: https://adc.bmj.com/content/early/2019/08/17/archdischild-2017-313375corr1
[2] Crawley E, Gaunt D, Garfield K, et al. Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 2018; 103:155-164
[3] Tuller D. Trial by error: the SMILE trial€™s undisclosed outcome-swapping. Virology Blog; Dec 13, 2017. Accessed at: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/
[4] Clinical trials: Hearings before the Science and Technology Committee, House of Commons; March 13, 2003. (Testimony of Fiona Godlee) Accessed at: https://publications.parliament.uk/pa/cm201213/cmselect/cmsctech/uc1054-i/uc105401.htm
[5] Weber W, Merino J, Loder E. Editorial: trial registration 10 years on. BMJ; 2015; 351. Accessed at: https://www.bmj.com/content/351/bmj.h3572
[6] Brown N. Editor’s note on correction to Crawley et al (2018). Archives of Disease in Childhood; 2019. Accessed at: https://adc.bmj.com/content/early/2019/08/17/archdischild-2017-313375ednote
[7] Bosely, S. Controversial Lightning Process ‘helps children with chronic fatigue syndrome.’ The Guardian; September 20, 2017. Accessed at: https://www.theguardian.com/society/2017/sep/20/controversial-lightning-process-helps-children-with-chronic-fatigue-syndrome-me
[8] Gregorowski A, Simpson J, Segal T. Child and adolescent chronic fatigue syndrome/myalgic encephalomyelitis; where are we now? Current Opinion in Pediatrics; 2019; 31: 462-46.
Comments are closed.