Trial By Error: BMJ Still “Looking Into” Lightning Process Paper

By David Tuller, DrPH

Two weeks ago I sent an e-mail to Sir Andrew Dillon and Professor Mark Baker alerting them of problems with the Lightning Process study published last year in Archives of Disease in Childhood. The two men are members of the NICE Guidance Executive and have a hand in the current process of developing what might be the agency€™s first ME/CFS guidance. (The 2007 guidance is for CFS/ME, and NICE has recently taken to calling the illness ME/CFS, like U.S. government agencies.)

My e-mail explained why the study€™s flaws meant it should not be cited as a reason to include the Lightning Process in the list of non-pharmacological treatments to be considered. I cc€™d several stakeholders in the ME/CFS guidance process as well as Fiona Godlee, editor in chief of The BMJ and editorial director at BMJ Company. I decided not to cc Nick Brown, the editor of Archives of Disease in Childhood. Dr Brown had been alerted to the problems in January and had given no signs of having done anything about the matter to date.

On Sunday, Dr Brown himself sent the following to all those included on my initial e-mail:

Dear Sir Andrew and Professor Baker

Dr Tuller’s e mail was forwarded to me by Fiona Godlee at the BMJ

We have previously acknowledged receipt of his complaints, are looking into them and, as we said, will respond fully when ready

Nick Brown

EiC, Archives of Disease in Childhood

**********

This morning, I sent the following to Dr Brown, and everyone else on the initial e-mail:

Dear Dr. Brown–

Thanks for your response to my concerns, as expressed in my recent e-mail to Sir Andrew, Professor Baker and others.

As I pointed out in that e-mail, on January 30th you received a letter from 21 experts. The letter criticized the study of the Lightning Process in children that Archives of Disease in Childhood published last year, noting that more than half the participants had been recruited as part of a feasibility study starting almost two years before trial registration. Publication of the trial therefore represented a clear violation of BMJ’s unequivocal policy not to publish trials in which participants were recruited before registration. Moreover, primary and secondary outcome measures were swapped based on the results of those feasibility trial participants–an obvious recipe for bias. Preventing just this sort of outcome-swapping is presumably one reason why BMJ and other major journals promulgated their strict requirements for prospective trial registration in the first place.

It has now been more than four months since we sent you that letter outlining the issues. During this extended period of time, you have not only not completed your review but have also refused my request to set a date for completing your review. It would literally take no more than a few minutes to determine the accuracy of my “complaints,” as you call them, so this delay is inexplicable. All you need to do is review the paper itself, the trial registration, and the research ethics committee application in which the investigators sought approval to include the feasibility study participants in the full trial while simultaneously swapping primary and secondary outcomes based on the early results. If you have trouble accessing any of these documents, I would be happy to forward them.

Perhaps I should remind you that the Lightning Process study involved not only children, but children suffering from a stigmatized illness. In other words, this was a vulnerable group among an already vulnerable population. Given that salient factor as well as the seriousness of the documented methodological lapses, it is perplexing that you and other editors have not shown any apparent urgency in pursuing this matter. Because of the paper your journal published, practitioners of the Lightning Process now have bragging rights that their training program has proven effective for kids in a randomized trial. The widespread media coverage has likely encouraged other desperate parents to subject their children to this regimen. In failing to take prompt steps to resolve the issue, Archives and BMJ are demonstrating a surprisingly cavalier attitude toward the interests and wellbeing of these families.

As I have noted previously, the Lightning Process investigators chose not to mention in the Archives paper that more than half the children were recruited before trial registration as part of a feasibility study. The investigators also did not mention that the outcome measures were swapped halfway through based on these feasibility study results. The omission of such critical information from the trial report is deeply disturbing. Not only did the investigators bias their results by swapping primary and secondary outcomes, they then failed to disclose these key mid-trial changes to readers of the Archives paper.

So why, after more than four months of investigation, have you still not managed to confirm that the Lightning Process study violated BMJ’s own policy about trial registration? Why have you not acknowledged that the investigators biased their findings by swapping outcome measures based on results from more than half the participants? Why have you not informed your readers of these methodological anomalies? Why has Dr Godlee herself not stepped in to make the appropriate determinations, since you and your editorial team appear unable to handle your responsibilities in a timely manner? Would you agree that this study is likely to impact medical treatment and public policy involving children, and that addressing the issues quickly is therefore critical?

In your e-mail, you noted that you will “respond fully” when “ready.” Apparently you are not only not yet ready to respond fully but are not even ready to set a deadline for responding fully. From my perspective, it is unclear when or whether you or Dr. Godlee will ever respond fully, or at least with the integrity and honesty expected of editors at leading medical journals. That’s why I decided I had an obligation to warn those involved in developing the new NICE guidance that the Lightning Process study’s methodological lapses rendered it unacceptable for consideration in any policy-making process.

I will continue to make that argument as necessary to health agency executives, members of parliament, government officials, journalists and other relevant parties until Archives of Disease in Childhood and BMJ stop stonewalling and finally address the self-evident deficiencies of the Lightning Process trial. I will continue to cc Dr Godlee on such communications.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley