By David Tuller, DrPH
Alan Montgomery is a professor of medical statistics and clinical trials at the University of Nottingham€™s School of Medicine. He is also the senior author of the Lightning Process study published in 2017 in Archives of Disease in Childhood, a BMJ journal. Professor Montgomery formerly worked at University of Bristol, along with Professor Esther Crawley and the other co-authors of the Lightning Process study.
As I documented more than a year ago, the Lightning Process study is a methodological and ethical mess that violates core scientific principles. Despite the potential public policy impact of the reported findings, Archives of Disease in Childhood has so far failed to fully resolve the issues. Given the journal€™s inability or unwillingness to take the prompt action necessary to protect this vulnerable patient group from bad research, I decided to reach out to Professor Montgomery directly. Considering his credentials, it is perplexing that he agreed to be senior author of the paper, since it improperly identified the study as prospective and failed to mention major outcome-swapping half-way through.
I e-mailed the following letter to Professor Montgomery on Sunday evening (California time), with the subject heading “Retrospective Registration and Outcome-Swapping in Lightning Process Trial.” I cc€™d Dr Nick Brown, editor-in-chief of Archives of Disease in Childhood, and Dr Fiona Godlee, editorial director of BMJ. In addition, I cc€™d three members of Parliament who have expressed dismay about the poor quality of research in this field and are seeking a parliamentary debate on the issue: Carol Monaghan MP, Darren Jones MP and Nicky Morgan MP. Finally, I cc’d Adam Lowe, Sally Burch and Dr Charles Shepherd, three members of the committee that is developing the ME/CFS guidance for the National Institute for Health and Care Excellence.
**********
Dear Professor Montgomery:
I am a journalist and public health researcher at the University of California, Berkeley. For the last few years, I have been investigating problematic studies in the ME/CFS field. Much of my work has appeared on Virology Blog, a site hosted by a Columbia University microbiologist, Professor Vincent Racaniello. This open letter will be posted there as well.
In September, 2017, Archives of Disease in Childhood published a study called Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial. You are the senior author of that paper; your eight co-authors are all affiliated with the University of Bristol.
The Lightning Process study was an open-label trial relying on self-reported outcomes, a design likely to generate results fraught with bias. Beyond that, as I have documented repeatedly during the past year on Virology Blog and in letters to BMJ editors, the trial featured serious methodological and ethical violations.
In this case, a pediatric Lightning Process feasibility trial was extended into the full study even as primary and secondary outcome measures were swapped based on the early results. These major changes were not mentioned in the Archives paper, although they allowed the findings to be presented in a more favorable light than would otherwise have been possible. The changes are only apparent from a review of other documents, such as the trial registration, the feasibility trial protocol and the full-trial protocol.
My specific concerns include the following:
*The study was registered as a prospective trial. But 56 out of 100 participants were recruited as part of the feasibility trial, starting almost two years before the registration date. For unexplained reasons, the Archives paper did not mention this fact. Like all major publishers of medical journals, BMJ has a longstanding policy of not considering any clinical trials in which participants were recruited before registration. In considering and then publishing the Lightning Process paper, Archives of Disease in Childhood violated its own stringent guidelines.
*In the feasibility trial protocol, school attendance at six months was the primary outcome, and self-reported physical function at six months was a secondary outcome. After the collection of data for the feasibility trial, these outcome measures were swapped for the full trial, school attendance at six months became a secondary outcome, and self-reported physical function at six months became the primary outcome. As with the trial registration issue, the Archives paper did not disclose the outcome-swapping that occurred after more than half the sample had been recruited as participants in the feasibility trial.
*As its main finding, the Archives paper highlighted the reported benefits in the Lightning Process group for self-reported physical function at six months, the new primary outcome. Self-reported school attendance at six months, now a secondary outcome, yielded null results and received scant attention in the Archives paper. Thus, the outcome-swapping biased the findings and improved the apparent success of the intervention. Had the measures not been switched, the paper would have had to report null results for the primary outcome.
*In both the feasibility trial protocol and the full-trial protocol, the researchers promised to seek official school attendance records as a way of vetting the accuracy of self-reported school attendance. The Archives paper did not provide official school attendance records, even though it highlighted benefits at 12 months in self-reported school attendance. No explanation for the absence of these official school attendance records was provided, leaving two possibilities: Either they were not obtained, or they were obtained and failed to confirm the benefits in self-reported school attendance at 12 months.
*Not surprisingly, the media coverage of the Archives paper focused on the reported benefits in the Lightning Process arm for the revised primary outcome of self-reported physical function at six months. The null results reported for school attendance at six months, the original primary outcome, were largely ignored. The undisclosed outcome-swapping thus ensured that the reported findings received more positive media coverage than they otherwise would have.
Transparency is essential in scientific research. Yet the Archives paper failed to include salient information about trial recruitment and registration, the decision to swap outcomes midway through, and the disposition of the official school attendance records, all factors that impact how the data should be interpreted. Last June, a published editor€™s note suggested that the concerns raised about the study were valid and that the paper€™s fate was under ongoing consideration. Seven months later, no further actions have been announced, even though a thorough assessment of the matter requires only a brief review of the relevant documentation.
It is worth reiterating that the Lightning Process is a pseudo-scientific intervention combining life-coaching, neuro-linguistic programming and osteopathy. Participants are taught that they can overcome illness by controlling and changing their thought patterns. Lightning Process practitioners have declared€“without citing legitimate evidence€“that they can successfully treat multiple sclerosis, eating disorders and other serious conditions. Government regulators have admonished some practitioners for making misleading claims.
Phil Parker, the creator of the Lightning Process, previously taught a course on how to heal people with divination medicine cards and tarot. The archived website for this course explains that divination is useful in creating a strong connection with healing/spirit guides. The course also featured lessons in the use of auras for diagnosis of a client€™s problems and in how to prepare a space appropriately so that any energy polluting the room will not interfere with the work you are doing.
Here is Phil Parker€™s biography from the same archived website:
Phil Parker is already known to many as an inspirational teacher, therapist, healer and author. His personal healing journey began€¦whilst working with his patients as an osteopath. He discovered that their bodies would suddenly tell him important bits of information about them and their past, which to his surprise turned out to be factually correct! He further developed this ability to step into other people€™s bodies over the years to assist them in their healing with amazing results. After working as a healer for 20 years, Phil Parker has developed a powerful and magical program to help you unlock your natural healing abilities. If you feel drawn to these courses then you are probably ready to join.
Professor Montgomery, this situation has dragged on for too long. The facts here are not in dispute. I urge you, as senior author of the Lightning Process study, to take responsibility for the paper€™s missteps. Did any of your co-authors recognize or point out that the paper presented a misleading and inaccurate account of the research? If not, why not?
Given that the trial was improperly registered as prospective, Archives of Disease in Childhood should not have published the paper in the first place. The appropriate action, therefore, is to retract it along with a full explanation of how these methodological and ethical anomalies occurred. You and your co-authors should also apologize to any families that have spent hundreds of pounds for the Lightning Process based on your reported positive findings.
I have cc€™d Dr Nick Brown, editor-in-chief of Archives of Disease in Childhood, and Dr Fiona Godlee, editorial director of BMJ. In addition, I have cc€™d three members of Parliament who have expressed dismay about the poor quality of research in this field and are seeking a parliamentary debate on the issue: Carol Monaghan MP, Darren Jones MP and Nicky Morgan MP. Finally, because the National Institute on Health and Care Excellence might consider the Lightning Process during its ongoing efforts to develop a guidance for ME/CFS, I have cc’d three members of the committee tasked with the assignment.
Thank you for your prompt attention to this matter.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.