By David Tuller, DrPH
This morning I sent the following letter to Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ. I cc-d Carol Monaghan MP, Darren Jones MP, and Nicky Morgan MP.
**********
Dear Dr. Godlee,
More than a dozen years ago, a group of leading medical journals agreed that requiring clinical trials to be prospectively registered could reduce publication bias and selective outcome reporting–and thus help to protect research integrity and credibility. In 2013, you testified before the House of Commons Science and Technology Committee about the importance of this collective initiative.
Here is part of your testimony: When the major journals first said in 2005 that they would not only publish but only peer review and consider randomised trials that had been prospectively registered, it sent a strong message€¦As far as we are aware, in the last two years we have not published any trial that has not been prospectively registered.
Here is BMJ€™s current policy, as described on the organization€™s website: In accordance with the ICMJE [International Committee of Medical Journal Editors] Recommendations, BMJ will not consider reports of clinical trials unless they were registered prospectively before recruitment of any participants.
And earlier this year, the former editor-in-chief of BMJ Open, Trish Groves, tweeted the following: Prospective registration of #clinicaltrials isn€™t red tape: it€™s done for sound ethical and scientific reasons.
These statements have a logical corollary. If trials that should not have been published because they were not prospectively registered somehow managed to slip through the editorial cracks, they should presumably be retracted once the facts were confirmed. Otherwise, unreliable results from such studies could still be touted as support for unproven and potentially harmful treatments.
Yet a high-impact BMJ journal, Archives of Disease in Childhood, is currently in breach of this implicit commitment. Since last December, I have described this matter in multiple blog posts and in open letters to both you and the journal€™s editor-in-chief, Nick Brown. (I am cc-ing Carol Monaghan MP and Darren Jones MP on this letter; they both currently sit on the Science and Technology Committee. I am also cc-ing Nicky Morgan MP, who has an interest in the issue.)
To recap: Last year, Archives of Disease in Childhood published a study of the Lightning Process as a treatment for pediatric chronic fatigue syndrome. In the paper, the investigators presented the trial as if it were prospective even though it was not. They also did not disclose that they engaged in outcome-switching mid-way through the study. Notwithstanding these documented flaws, the paper remains in the literature, suggesting that in this case the journal might be prioritizing reputational interests over the health and well-being of children with a stigmatizing illness.
The Lightning Process is a pseudo-scientific intervention that combines life-coaching, neuro-linguistic programming and osteopathy. Participants in the programme are taught that they can overcome illness by controlling and changing their thought patterns. Lightning Process practitioners have declared–without citing legitimate evidence–that they can successfully treat multiple sclerosis, eating disorders and other serious conditions. Government regulators have admonished some practitioners for making misleading claims.
Phil Parker, the creator of the Lightning Process, previously taught a course on how to heal people with divination medicine cards and tarot. The archived website for this course explains that divination is useful in creating a strong connection with healing/spirit guides. The course also featured lessons in the use of auras for diagnosis of a client€™s problems and in how to prepare a space appropriately so that any energy polluting the room will not interfere with the work you are doing.
Here is Phil Parker€™s biography from the same archived website:
Phil Parker is already known to many as an inspirational teacher, therapist, healer and author. His personal healing journey began…whilst working with his patients as an osteopath. He discovered that their bodies would suddenly tell him important bits of information about them and their past, which to his surprise turned out to be factually correct! He further developed this ability to step into other people€™s bodies over the years to assist them in their healing with amazing results. After working as a healer for 20 years, Phil Parker has developed a powerful and magical program to help you unlock your natural healing abilities. If you feel drawn to these courses then you are probably ready to join.
In the situation under review, the investigators recruited 56 pediatric participants as part of a Lightning Process feasibility study. The regional research ethics committee approved the request to extend the feasibility study into a full trial. At the same time, the committee also allowed the investigators to swap primary and secondary outcomes. The investigators registered the full trial and recruited more children, obtaining a total sample of 100 participants.
In approving these steps, the research ethics committee effectively enabled the kind of selective outcome reporting that the requirement for prospective registration was supposed to prevent. With the outcome swap and the 44 added participants, the investigators reported positive results for the revised primary outcome of self-reported physical function at six months–a finding that received widespread media attention. Yet they reported null results for the original primary outcome of school attendance at six months. Since this was now a secondary outcome, the findings were essentially ignored.
Moreover, the paper itself was written with a singular lack of transparency. The investigators did not explain that they recruited 56 % of the participants before trial registration and that they swapped outcome measures based on the early findings. It is unclear why experienced researchers would decide to omit these key details from the published report.
It should also be noted that even without these anomalies, the study would not have been able to provide much useful information. Like many of the trials in this field of research, it was an open-label study relying on subjective outcomes. When a trial combines these two traits, the results are at high risk of bias and cannot be considered authoritative.
In this case, school attendance would have been a fully objective outcome had the investigators not again engaged in a form of outcome-switching. Although they reported null results for school attendance at six months, they reported positive results for school attendance at 12 months. In their protocol, the investigators had promised to vet self-reported accounts of school attendance, which can suffer from memory bias, against official school attendance records. Yet the published school attendance results were based solely on self-reported accounts of school attendance.
The investigators did not include any data from the official school attendance records in the published paper. Nor did they disclose whether they accessed these official records and, if so, whether these records confirmed or contradicted the self-reported accounts of school attendance. Absent these salient facts, it is impossible to take the self-reported 12-month results at face value.
Despite these violations of core BMJ principles, Archives of Disease in Childhood has not retracted the study. Instead, the journal has added an editor€™s note that is not visible from the paper’s home page but is hidden away in the responses section of the site. Undoubtedly, most readers, including desperate parents seeking answers–have not seen it.
This editor€™s note, posted in June, appears to acknowledge the validity of the concerns. It explains that Archives of Disease in Childhood has reviewed its editorial processes and has received “clarifications” from the investigators. The note suggests that the matter is under editorial consideration. Since it only takes a few minutes to verify the relevant documentation, it is hard to understand why no further action has yet been taken, especially given the paper€™s possible impact on pediatric care and public health policy.
What more information does Archives of Disease in Childhood need to end the protracted process of editorial consideration? Five years ago, Dr Godlee, you spoke forthrightly to members of Parliament about defending science with a zero-tolerance policy on the issue of prospective trial registration. But the undecided fate of the Lightning Process paper suggests that members of your editorial team do not share much enthusiasm for the high standard you articulated.
To date, the investigators and their institution have not cleaned up the mess they have created. That failure should not preclude Archives of Disease in Childhood from retroactively applying the requirement for prospective registration and from issuing a transparent acknowledgement of its own lapses in oversight. The published paper has provided, and still provides, bragging rights to Lightning Process practitioners everywhere. Given the methodological and ethical aberrations that mar the study and render its conclusions unsupportable, the journal’s continuing inability or unwillingness to ensure the accuracy of the scientific literature is unacceptable.
Best, David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.