By David Tuller, DrPH
In July, I sent Dr Fiona Godlee, editorial director of BMJ, a letter signed by 55 experts about her company’s perplexing decision to republish the originally reported–and unreliable–findings from the trial of the Lightning Process. She did not respond. This morning I sent the letter again, with more individual signatories along with dozens of patient and advocacy groups.
**********
Dear Dr Godlee—
We are writing about the correction appended in June [1] to the pediatric study of the Lightning Process conducted by investigators from the University of Bristol and published by Archives of Disease in Childhood in September, 2017. The study appeared under the following title: “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.” [2]
As Virology Blog documented shortly after publication [3], the investigators recruited more than half the sample before trial registration as part of a feasibility study, swapped primary and secondary outcomes after gathering data from these early participants, and did not disclose these salient details in the paper. With the correction and an accompanying editor’s note, Archives of Disease in Childhood has now acknowledged the validity of Virology Blog’s criticisms. Yet the journal has allowed the investigators to reassert their main conclusion–that the Lightning Process is “effective” in treating pediatric illness. This decision is untenable.
The Lightning Process is a commercial program combining life-coaching, neuro-linguistic programming and positive affirmations. Participants are taught that they can overcome illness by controlling and changing their thought patterns. Lightning Process practitioners have declared that they can successfully treat multiple sclerosis, eating disorders and other serious conditions. Government regulators have admonished practitioners for making misleading claims.
Beyond concerns about the Lightning Process itself, the Bristol study was an open-label trial with self-reported outcomes, a combination of design features that can lead to significant bias. Furthermore, in violating core methodological and ethical principles of scientific inquiry, the investigators also breached BMJ’s own strict policy against publishing any trials in which participants have been recruited before registration. Since all major medical journals adopted this policy in 2005, experienced investigators, including members of the Bristol team, should have been well aware of it.
BMJ has been a leader in the movement to require prospective registration as a condition for trials to be considered for publication. In 2013, in testimony before the House of Common’s Science and Technology Committee, you touted BMJ’s success in implementing a zero-tolerance approach to such trials. “As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered,” you stated, a claim you could not accurately repeat today. [4] A 2015 editorial in The BMJ called the prospective registration policy “the single most valuable tool we have to ensure unbiased reporting of research studies.” [5]
Given BMJ’s longstanding position that prospective registration is essential for safeguarding the integrity of the medical literature, it is unclear why you are now willing to exempt the Lightning Process study from this requirement. The editor’s note posted with the correction declared that lack of prospective registration was not grounds for retraction. [6] No explanation or justification was provided for this determination, which appears to be incompatible with the zero-tolerance approach you advocated in your parliamentary testimony.
As you certainly know, a key goal of prospective registration is to prevent selective outcome reporting. Yet selective outcome reporting is exactly what occurred in the Lightning Process study when the investigators swapped primary and secondary outcomes mid-way through. In the published paper, the investigators reported positive results for the revised primary outcome of self-reported physical function at six months, which had initially been designated a secondary outcome. In contrast, they reported null results for the original primary outcome of school attendance at six months, which by the end had been demoted to a secondary outcome.
The swap thus ensured that the investigators were able to report positive rather than null results for their official primary outcome. Not surprisingly, this change had a major impact on how the information was presented in Archives of Disease in Childhood and interpreted by news organizations. The positive findings for the primary outcome that was selected after more than half the participants had provided data were prominently highlighted in the published paper. These findings received widespread media attention [7] and have recently been cited in a systematic review as evidence that the intervention is “effective.” [8] The null findings for the original primary outcome were ignored.
The editor’s note explains that the journal examined the outcome-swapping issue by “seeking assurance from the authors that the change in primary outcome was not influenced by (positive) findings in the feasibility phase.” This statement is perplexing. Individuals subject to potential bias are not generally considered impartial and authoritative arbiters of whether this potential bias has influenced their decision-making. Study design is supposed to seek to minimize bias precisely because humans tend to be blind to their own biases. Reputable science in high-impact journals should not have to rely on investigators’ “assurance” that they have resisted the natural temptation to let their perceived interests guide their selection of primary outcomes.
Moreover, in relation to any assurances provided by the Bristol investigators, it is worth remembering that they withheld important information about trial registration and outcome-swapping from their public version of events. Archives of Disease in Childhood would not have published the paper in the first place except for two major failures: 1) The investigators’ failure to provide an accurate account of how they conducted the trial; and 2) The journal’s failure to detect disqualifying flaws, despite its obligation to subject the paper to rigorous peer review and editorial oversight.
Archives of Disease in Childhood has now re-published positive primary results generated by outcome-swapping in a study that did not meet a strict publication requirement specifically intended to prevent outcome-swapping and other kinds of selective outcome reporting. In doing so, the journal has rewarded the Bristol investigators for their lack of candor and their methodological missteps and has contradicted BMJ’s and your own past statements about the critical importance of prospective registration.
In this case, BMJ’s actions are scientifically and ethically indefensible. They are also potentially harmful to the health and well-being of children. Thank you for your attention to this troubling matter.
This letter is being cc’d to many of the signatories as well as to others with an interest in this issue, including: several members of the committee from the National Institute for Health and Care Excellence currently developing new guidelines for ME/CFS; the director of legal services at the University of Bristol; the chair and vice chair of the CFS/ME Research Collaborative; the senior author of the Lightning Process study; the senior author of a review that cited the Lightning Process study and the editor of the journal that published that review; and a few journalists.
Best–
Dharam V. Ablashi, DVM, MS, Dip Bact
Scientific Director
HHV-6 Foundation
Santa Barbara, California, USA
Former Senior Investigator
National Cancer Institute
National Institutes of Health
Bethesda, Maryland, USA
Raeka Aiyar, PhD
Senior Director of Scientific Outreach
New York Stem Cell Foundation Research Institute
New York, New York, USA
Michael Allen, PhD
Clinical Psychologist (retired)
San Francisco, California, USA
Chris Armstrong, PhD
Science Liaison
Open Medicine Foundation
Agoura Hills, California, USA
Visiting Scholar
Stanford University
Stanford, California, USA
James N. Baraniuk, MD
Professor of Medicine
Georgetown University
Washington, DC, USA
Lisa F. Barcellos, MPH, PhD
Professor of Epidemiology
School of Public Health
California Institute for Quantitative Biosciences
University of California, Berkeley
Berkeley, California, USA
Lucinda Bateman, MD
Medical Director
Bateman Horne Center
Salt Lake City, Utah, USA
Alison C. Bested, MD, FRCPC
Clinical Director
Institute for Neuro-Immune Medicine
Associate Professor
Nova Southeastern University
Fort Lauderdale, Florida, USA
Charlotte Blease, PhD
Fulbright and Marie Curie Research Fellow
General Medicine and Primary Care
Beth Israel Deaconess Medical Center
Harvard Medical School
Boston, Massachusetts, USA
Molly Brown, PhD
Assistant Professor
Department of Psychology
DePaul University
Chicago, Illinois, USA
Robin Callender Smith, PhD
Professor of Media Law
Centre for Commercial Law Studies
Queen Mary University of London
Barrister and Information Rights Judge
London, England, UK
Joan Crawford, CPsychol, CEng, CSci, MA, MSc
Chartered Counselling Psychologist
Chronic Pain Management Service
St Helens Hospital
St Helens, England, UK
Janet L. Dafoe, PhD
Child Psychologist in Private Practice
Palo Alto, California, USA
Todd E. Davenport, PT, DPT, MPH, OCS
Professor and Program Director
Department of Physical Therapy
Thomas J. Long School of Pharmacy & Health Sciences
University of the Pacific
Stockton, California, USA
Workwell Foundation
Ripon, California, USA
Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Stanford, California, USA
Simon Duffy, PhD, FRSA
Director
Centre for Welfare Reform
Sheffield, England, UK
Jonathan C.W. Edwards, MD
Emeritus Professor of Medicine
University College London
London, England, UK
Valerie Eliot Smith
Barrister and Visiting Scholar
Centre for Commercial Law Studies
Queen Mary University of London
London, England, UK
Margaret C. Fernald, PhD
Clinical Child Psychologist
Clinical Associate of Psychology
University of Maine
Orono, Maine, USA
Kenneth J. Friedman, PhD
Associate Professor of Physiology and Pharmacology (retired)
New Jersey Medical School
University of Medicine and Dentistry of New Jersey
Newark, New Jersey, USA
Robert F. Garry, PhD
Professor of Microbiology and Immunology
Tulane University School of Medicine
New Orleans, Louisiana, USA
Claudia Gillberg, PhD
Fellow, Centre for Welfare Reform
Sheffield, England, UK
Senior Research Associate
National Centre for Lifelong Learning
Jonkoping University
Jonkoping, Sweden
Rebecca Goldin, PhD
Professor of Mathematics
George Mason University
Fairfax, Virginia, USA
Alan Gurwitt, MD
Psychiatrist in Private Practice (retired)
Associate Clinical Professor
Yale Child Study Center (retired)
New Haven, Connecticut, USA
Associate Clinical Professor
University of Connecticut Dept of Psychiatry (retired)
Storrs, Connecticut, USA
Lecturer, Harvard Medical School (retired)
Boston, Massachusetts, USA
Paul Guyre, PhD
Emeritus Professor Micro Immunology
Geisel School of Medicine
Dartmouth
Hanover, New Hampshire, USA
Geoffrey Hallmann, LLB, DipLegPrac, DipFinPrac
Specialist in Disability and Compensation Law (retired)
Lismore, New South Wales, Australia
Brian M. Hughes, PhD, FPsSI
Professor of Psychology
National University of Ireland, Galway
Galway, Ireland
Leonard A. Jason, PhD
Professor of Psychology
DePaul University
Chicago, Illinois, USA
Jonathan Kerr, MD, PhD, FRCPath
Consultant in Microbiology
West Suffolk Hospital
Bury St Edmunds, England, UK
Maureen Hanson, PhD
Liberty Hyde Bailey Professor
Department of Molecular Biology and Genetics
Cornell University
Ithaca, New York, USA
Keith Kahn-Harris, PhD
Associate Lecturer in Sociology
Birkbeck, University of London
London, England, UK
Mady Hornig, MD
Associate Professor of Epidemiology
Mailman School of Public Health
Columbia University
New York, New York, USA
Jon D. Kaiser, MD
Clinical Faculty
Department of Medicine
University of California Medical School
San Francisco, California, USA
Nancy Klimas, MD
Director, Institute for Neuro-Immune Medicine
Nova Southeastern University
Director, Miami VA Medical Center GWI and CFS/ME Program
Miami, Florida, USA
Eliana M. Lacerda, MD, MSc, PhD
Clinical Assistant Professor
International Centre for Evidence in Disability
Faculty of Infectious and Tropical Diseases
London School of Hygiene & Tropical Medicine
London, England, UK
Bruce Levin, PhD
Professor of Biostatistics
Columbia University
New York, New York, USA
Rogier Louwen, PhD
Assistant Professor
Department of Medical Microbiology and Infectious Diseases
Erasmus University Medical Center
Rotterdam, The Netherlands
Steven Lubet, JD
Williams Memorial Professor of Law
Northwestern University Pritzker School of Law
Chicago, Illinois, USA
Sonya Marshall-Gradisnik
Professor of Immunology
Menzies Health Institute Queensland
Co-Director
National Centre for Neuroimmunology and Emerging Diseases
Griffith University
Gold Coast, Queensland, Australia
Marlon Maus, MD, DrPH, FACS
Director, DrPH Program
School of Public Health
University of California, Berkeley
Berkeley, California, USA
Neil R. McGregor, BDS, MDSc, PhD
Clinical Associate Professor
Faculty of Medicine, Dentistry and Health Sciences
University of Melbourne.
Melbourne, Victoria, Australia
Patrick E. McKnight, PhD
Professor of Psychology
George Mason University
Fairfax, Virginia, USA
Karl Morten, PhD
Laboratory Manager
Medical Services Division
University of Oxford
Oxford, England, UK
Nina Muirhead, BMBCh
Dermatologic Surgeon
Buckinghamshire NHS Trust
Buckinghamshire, England, UK
Heidi Nicholl, PhD
Chief Executive Officer
Emerge Australia
Melbourne, Victoria, Australia
Steve Olson, MD
Family Physician
Large Medical Group Executive and Regional Physician Director
Oakland, California, USA
Elisa Oltra, PhD
Professor of Molecular and Cellular Biology
Catholic University of Valencia School of Medicine
Valencia, Spain
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
New York, New York, USA
Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley
Berkeley, California, USA
Peter C. Rowe, MD
Professor of Pediatrics
Johns Hopkins University School of Medicine
Baltimore, Maryland, USA
Ola Didrik Saugstad, MD, PhD
Professor Emeritus
Division of Paediatric and Adolescent Medicine
University of Oslo
Oslo, Norway
Adjunct Professor of Pediatrics
Feinberg School of Medicine
Northwestern University
Chicago, Illinois, USA
David Scales, MD, PhD
Assistant Professor of Medicine
Division of Hospital Medicine
Weill Cornell Medicine
New York, New York, USA
Michael Scott, PhD
Clinician and Researcher
Psychological Therapies Unit
Liverpool, England, UK
Charles Shepherd, MBBS
Honorary Medical Adviser to the ME Association
Buckingham, England, UK
Donald R. Staines, MBBS, MPH, FAFPHM, FAFOEM
Clinical Professor
Menzies Health Institute Queensland
Co-Director
National Centre for Neuroimmunology and Emerging Diseases
Griffith University
Gold Coast, Queensland, Australia
Philip B. Stark, PhD
Professor of Statistics
University of California, Berkeley
Berkeley, California, USA
Eleanor Stein, MD, FRCP(C)
Psychiatrist in Private Practice
Assistant Clinical Professor
University of Calgary
Calgary, Alberta, Canada
Leonie Sugarman, PhD
Emeritus Associate Professor of Applied Psychology
University of Cumbria
Carlisle, England, UK
John Swartzberg, MD
Clinical Professor Emeritus
School of Public Health
University of California, Berkeley
Berkeley, California, USA
Susan Taylor-Brown, PhD
Clinical Professor of Pediatrics (retired)
Golisano Children’s Hospital
University of Rochester Medical Center
Rochester, New York, USA
Ronald G. Tompkins, MD, ScD
Summer M Redstone Professor of Surgery
Harvard Medical School
Boston, Massachusetts, USA
Samuel Tucker, MD
Assistant Clinical Professor of Psychiatry (retired)
University of California, San Francisco
San Francisco, California, USA
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
University of California, Berkeley
Berkeley, California, USA
Rosamund Vallings, MNZM, MBBS
General Practitioner
Auckland, New Zealand
Linda van Campen, MD
Cardiologist
Stichting Cardiozorg
Hoofddorp, The Netherlands
J. Mark VanNess, PhD
Professor and Chair
Health and Exercise Science
University of the Pacific
Stockton, California, USA
Mark Vink, MD
Family Physician
Soerabaja Research Center
Amsterdam, The Netherlands
Frans Visser, MD
Cardiologist
Stichting Cardiozorg
Hoofddorp, The Netherlands
Tony Ward, MA (Hons), PhD, DipClinPsyc
Registered Clinical Psychologist
Professor of Clinical Psychology
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
Adjunct Professor, School of Psychology
University of Birmingham
Birmingham, England, UK
Adjunct Professor, School of Psychology
University of Kent
Canterbury, England, UK
William Weir, FRCP
Infectious Disease Consultant
London, England, UK
John Whiting, MD
Specialist Physician in Private Practice
Brisbane, Queensland, Australia
Carolyn Wilshire, PhD
Senior Lecturer
School of Psychology
Victoria University of Wellington
Wellington, New Zealand
**********
#MEAction UK
#MEAction USA
12ME
Belgium
25% ME GROUP – Support for severe ME sufferers
UK
Action CIND
Canada
Action for ME
UK
American ME and CFS Society
USA
ANZMES (Associated New Zealand ME Society Inc)
New Zealand
Central London ME Group
UK
Chester ME Self-help (MESH)
UK
Deutsche Gesellschaft für ME/CFS | German Association of ME/CFS
Germany
Dr Myhill’s MAIMES Campaign
UK
Emerge Australia
Australia
European ME Alliance (EMEA)
Belgium ME/CFS Association
Belgium
Czech Republic ME/CFS Association
Czechoslovakia
ME FORENINGEN
Denmark
Suomen ME/CFS-yhdistys
Finland
Suomen lääketieteellinen ME/CFS-Yhdistys ry
Finland
ASFC РAssociation Fran̤aise du Syndrome de Fatigue Chronique
France
Fatigatio e.V.
Germany
Icelandic ME Association
Iceland
Irish ME Trust
Ireland
Associazione CFS Veneto
Italy
Het Alternatiet
The Netherlands
Norges ME-forening
Norway
Liga SFC
Spain
RME Riksföreningen för ME-patienter
Sweden
Verein ME/CFS Schweiz
Switzerland
Hope 4 ME & Fibro Northern Ireland
UK
Invest in ME Research
UK
FNC ME/CFS Association
Australia
Global Advocates 4 Myalgic Encephalomyelitis (GAME) Canada (*Added to list on Nov 22 after letter was sent to Dr Godlee)
Groep ME-DenHaag
The Netherlands
Lost Voices Foundation
Germany
Massachusetts ME/CFS & FM Association
USA
ME Advocacy Ireland
Ireland
ME Association
UK
ME Research UK
UK
ME/CFS Australia Ltd.
Australia
ME/CFS & FM Association NSW
Australia
ME/CFS and Lyme Association of WA, Inc.
Australia
ME/CFS South Australia Inc
Australia
ME/CFS/FM Support Association Qld Inc.
Australia
ME/CVS Stichting Nederland (ME/CVS Foundation Netherlands)
The Netherlands
ME/cvs Vereniging
The Netherlands
ME/FM Society of British Columbia
Canada
MEadvocacy.org
USA
Millions Missing Canada
Canada
Minnesota ME/CFS Alliance
USA
National CFIDS Foundation, Inc.
USA (*Corrected on Nov 22 from “CFIDS Foundation, Inc.”)
National ME/FM Action Network
Canada
New Jersey ME/CFS Association, Inc.
USA
North London ME Network
UK
Open Medicine Foundation
USA
Opposing MEGA
UK
(*Added to list after letter sent to Dr Godlee)
Oxfordshire ME Group for Action
UK
(*Added to list after letter sent to Dr Godlee)
Pandora Org
USA
Science for ME
UK (online forum with members from many countries)
Sheffield ME and Fibromyalgia Group
UK
Solve ME/CFS Initiative
USA
Steungroep ME en Arbeidsongeschiktheid
The Netherlands
Suomen lääketieteellinen ME/CFS-yhdistys ry
Finland
The York ME Community
UK
The Young ME Sufferers Trust
UK
Wake-Up Call Beweging
Belgium
WAMES – Welsh Association of ME & CFS Support
UK
Wisconsin ME and CFS Association, Inc
USA
**********
[1] Crawley E, Gaunt D, Garfield K, et al. Notice of correction and clarification: Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 2019. Accessed at: https://adc.bmj.com/content/early/2019/08/17/archdischild-2017-313375corr1
[2] Crawley E, Gaunt D, Garfield K, et al. Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. Archives of Disease in Childhood; 2018; 103:155-164
[3] Tuller D. Trial by error: the SMILE trial’s undisclosed outcome-swapping. Virology Blog; Dec 13, 2017. Accessed at: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/
[4] Clinical trials: Hearings before the Science and Technology Committee, House of Commons; March 13, 2003. (Testimony of Fiona Godlee) Accessed at: https://publications.parliament.uk/pa/cm201213/cmselect/cmsctech/uc1054-i/uc105401.htm
[5] Weber W, Merino J, Loder E. Editorial: trial registration 10 years on. BMJ; 2015; 351. Accessed at: https://www.bmj.com/content/351/bmj.h3572
[6] Brown N. Editor’s note on correction to Crawley et al (2018). Archives of Disease in Childhood; 2019. Accessed at: https://adc.bmj.com/content/early/2019/08/17/archdischild-2017-313375ednote
[7] Bosely, S. Controversial Lightning Process ‘helps children with chronic fatigue syndrome.’ The Guardian; September 20, 2017. Accessed at: https://www.theguardian.com/society/2017/sep/20/controversial-lightning-process-helps-children-with-chronic-fatigue-syndrome-me
[8] Gregorowski A, Simpson J, Segal T. Child and adolescent chronic fatigue syndrome/myalgic encephalomyelitis; where are we now? Current Opinion in Pediatrics; 2019; 31: 462-46.
Comments are closed.