By David Tuller, DrPH
The trickle of letters from top experts to Dr Fiona Godlee about BMJ’s decision to republish Professor Esther Crawley’s Lightning Process study continues. The letters excoriate BMJ’s actions in this matter and urge Dr Godlee to retract the dung-heap otherwise known as “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial.”
I have previously posted six letters, here and here. Below are five more. They were e-mailed directly to Dr Godlee and cc’d to many of the 55 scientists, clinicians and other experts who signed Virology Blog’s recent open letter about the matter. The writers are: Dr Steve Olsen of Kaiser Permanente’s Northern California region; Professor Alison Bested of Nova Southeastern University in Florida; Professor Rebecca Goldin of George Mason University in Virginia; Professor Ronald Tompkins of Harvard Medical School; and Professor Brian Hughes of National University of Ireland, Galway.
The Lightning Process study was published online in September, 2017, by Archives of Disease in Childhood, a BMJ journal. After being alerted to the issue by some early questions about the study timeline raised on the Science For ME forum, I documented in December, 2017, that Professor Crawley and her colleagues recruited more than half the participants before trial registration, swapped outcome measures after collecting this early data, and failed to disclose these salient details in the published paper.
Instead of retracting the paper, Archives of Disease in Childhood recently republished the original findings, along with a massive correction/clarification and an editor’s note articulating laughable excuses for not retracting the paper. I obviously wasn’t the only one disturbed by BMJ’s blatant dereliction of editorial responsibility. Top experts were eager to sign the open letter to Dr Godlee. Now many of them are personally appealing to her to do the right thing, often in very strong language.
Will Dr Godlee listen?
*****
Dear Dr. Godlee,
I also signed the letter of August 27th regarding the inappropriate addition to the medical literature of BMJ€™s Lightning Process Study.
I have been a Family Physician for 30 years, and am a physician executive of the largest integrated medical group in the US, and board certified in medical informatics.
Do you honestly believe the Lightning Process has been validated as being an effective treatment for a severe multi system disease? If not then it should be retracted to prevent harm.
If you do believe it has been validated as an effective treatment and/or the article is not removed, then the BMJ no longer has credibility in the eyes of a growing number of clinicians. Further published studies, some of which may have value and merit, will be greatly discounted simply by being associated with BMJ.
Lastly, let me know if you would be willing to publish an article if I replaced Lightning Process with voodoo as the only change in methodology?
Steve Olson, MD
Family Physician
The Permanente Medical Group – Regional Director
Kaiser Permanente
Oakland, CA
*****
Dear Dr. Goldee,
Like many doctors and scientists, I signed a petition requesting that BMJ retract the article €œClinical and cost-effectiveness of the Lightning Process€¦€ I am writing to express again my grave concern about BMJ€™s publication practice as well as the results of the article.
As a mathematician and Director of STATS (a nonprofit involved with the accurate reporting of statistical aspects of medical research), I reviewed the article with great interest in its public face. Mainly, journalists and parents want to know whether the results €œare true€ and the methods €œare valid€. A modest effort at detangling the research methods makes evident damning weaknesses of the article.
The experimental design of this study seems like a textbook case of why the public shouldn€™t trust scientists. Post-facto decisions about outcome measurements (such as swapping primary and secondary outcomes), and pursuing explicitly forbidden scientific practices (such as recruiting participants trial registration through a feasibility study) increase the likelihood of false positives; when there are scientific and business motivations to do such switches, all credibility is lost.
I am frankly surprised that BMJ would be willing to risk its reputation to defend a study that looks on the outside like a corporate front. To title the piece a €œrandomized controlled trial€ is misleading as the participants were not randomly selected. To suggest that there is a clinical benefit when the data were so easily manipulated in ways we know about implies that brokers of scientific knowledge are no longer able to stem the poor science with corporate motives from entering the scientific mainstream. To suggest that the method is €œcost effective€ while the paper advertises the company€™s dubiously €œproven€ methods suggests to the public that the BMJ has motives that extend past the health of these children. Public suspicion of science is founded on poorly conceived €œresults” getting institutional endorsements such as a publication in BMJ.
Unfortunately for children who suffer from ME/CFE, BMJ still has a big impact on treatment protocols. You personally have the opportunity to stand for quality science, and BMJ should do so. Rescind the paper based on BMJ policies, admitted disinformation by the authors of the study, and data whose quality cannot be assessed due to collection methods. There is no harm to suggest that science can be done better by rescinding the paper, especially considering the data manipulation that has already been discovered. In contrast, the harm stemming from publishing results that are essentially cooked up will have a lasting impact both on children who need high quality research, on BMJ€™s reputation, and on public trust of medical science.
Rebecca Goldin, Ph.D.
Professor, Mathematical Sciences
George Mason University
Director, STATS
Sense About Science
*****
Dear Dr. Godlee,
I would ask that this article in the BMJ about the Lightning Process be retracted based on the incorrect methodology used to conduct this clinical trial.
I would also urge you to retract this article based on my experience as a specialist who has been treating patients for the past twenty-six years with Myalgic Encephalomyelitis/Chronic Fatigue Syndrome (ME/CFS).
One of my patients underwent this €œLightning Process€ in the hope of getting €œcured€ from her ME/CFS. Instead she had a severe relapse of her ME signs and symptoms and remained bedridden for months.
This program was very harmful to this patient. It gave her false hope and caused a serious relapse of her symptoms for many months.
I reviewed the article and immediately spread the word to all of my patients that they were not to have this harmful treatment.
The patients with ME/CFS deserve better treatment than this by us in the medical profession. We as physicians are here to help our patients heal and to €œabstain from all intentional wrong-doing and harm€. Hippocratic Oath
ME/CFS is a physical disease. The muscles and mitochondria in them are dysfunctional. As a result, patients have post-exertional fatigue. All the hopeful thinking in the world does not change this basic pathophysiology found in ME/CFS. Institute of Medicine Report 2015. https://www.ncbi.nlm.nih.gov/books/NBK284902/#sec_073
The BMJ can correct this situation by retracting this research article whose basis assumption is harmful to patients with ME/CFS.
I thank you for your consideration.
Sincerely,
Alison
Alison C. Bested MD FRCPC
Diplomate, American Board of Integrative Medicine
Chair, Integrative Medicine
Director, Student Research Development
Clinical Director, Institute for Neuro-Immune Medicine
Associate Professor
Dr. Kiran C. Patel College of Osteopathic Medicine
Center for Collaborative Research
Institute for Neuro-Immune Medicine
Nova Southeastern University
Davie, FL 33314
*****
Dear Dr. Godlee,
I am appalled at this situation. As a doctor of more than 40 years and a scientist, it is my opinion that a retraction of this article is in order. Ron
Ronald G. Tompkins, M.D., Sc.D.
Sumner Redstone Professor of Surgery
Harvard Medical School
Founding Director, Center for Surgery, Innovation & Bioengineering
Massachusetts General Hospital
Boston, MA 02114
*****
Dear Dr Godlee,
First of all, thank you for the work that you do as editor in chief of the BMJ. I do not at all mean this as faint praise. While my colleagues and I have expressed our concerns, I believe it is important to note that editing a journal is not always easy. Editors play a crucial part in the production line of science and your own contributions, and those of your editorial colleagues, need to be acknowledged.
Secondly, I feel obliged to add my voice to those others who have written to you about the paper “Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial” and, particularly, the associated correction that was published.
You will be familiar with the details of our concerns as outlined in previous emails.
From my own perspective, I have been investigating the issues around controversial therapies and human factors (i.e., error risks) in research for nearly two decades. You will be aware that expectancy effects and confirmation biases are very problematic in clinical therapy research in general. They create particular chaos with therapies that are controversial to begin with.
The so-called ‘Lightning Process’ is undoubtedly one such therapy. In fact, it comprises at a number of modalities that are normally classified as pseudosciences. It is based largely on so-called Neuro-Linguistic Programming (NLP), which is a completely discredited practice. The creator of the ‘Lightning Process’ is an advocate and provider of several pseudoscientific therapies (e.g., cranial osteopathy; applied kinesiology; hypnotherapy). All told, there is nothing to suggest that the ‘Lightning Process’ is a promising clinical modality. It has no scientific plausibility; it exists because commercially-minded providers of pseudoscientific treatments have successfully identified a market for it. In that regard, it occupies the same space as, say, crystal therapy.
In offering that description, I do not intend to be pejorative. I merely wish to point out that any hypothesis that the ‘Lightning Process’ is effective for any condition constitutes an “extraordinary claim.” By all means should extraordinary claims be tested. Studies of crystal therapy, for example, should be conducted. However, you will appreciate that in science, “extraordinary claims require extraordinary evidence.” Research purporting to establish the efficacy of the ‘Lightning Process’ needs to exhibit a truly compelling evidentiary standard. The claims made inherently challenge our mainstream understanding of human biology. If the ‘Lightning Process’ is in fact shown to work, we will need to re-write our physiology and neurology textbooks.
On the face of it, the evidence contained in this particular paper is far from extraordinary. In fact, as per your own published correction, the study bypassed the normal requirement for prospective registration. This immediately raises the red flag of reporting bias. The evidence accrued from the paper cannot be relied upon because the authors were able to select to report it, without pre-registering their intentions.
In short, confirmation bias was not prevented. Your request to the authors that they tell you whether or not they were biased is very insufficient.
The data produced in light of this procedural and methodological lapse by no means meet the standard of “extraordinary evidence”.
As I understand it, this paper would not have been published had you known about its irregularity at the time of initial review. Therefore it should not be published now. Therefore it should be retracted.
A serious problem with not retracting this paper is that its existence will strongly encourage other researchers to play fast and loose with pre-registration, in the knowledge that their findings can still be published in a BMJ journal. The paper will stand as a high-profile exemplar of ambiguous editorial commitment to scientific rigour.
Kind regards,
Professor Brian Hughes, PhD, FPsSI
School of Psychology
National University of Ireland, Galway
Comments are closed.