Trial By Error: NICE’s Consideration of the Lightning Process

By David Tuller, DrPH

Earlier this month, in advance of a stakeholder meeting, the U.K.’s National Institute for Health and Care Excellence released a draft scoping report. The document outlined the issues slated to be addressed by the committee selected to develop the new guidance for the illness NICE now calls ME/CFS. (The 2007 guidance referred to it as CFS/ME; the name switch represents, at least to me, a potentially positive albeit modest sign of shifting attitudes at NICE.)

According to follow-up reports from the Friday meeting, the discussion appeared to go reasonably well. A formal consultation period from June 21st to July 19th will provide an opportunity for people to submit further comments before the scoping report is finalized.

Smart stakeholders will hopefully be able to hold NICE to account on this matter. When I skimmed the draft scoping document, I wasn’t surprised to see graded exercise therapy and cognitive behavior therapy listed among the non-pharmaceutical interventions up for discussion. But the presence of the Lightning Process in the list caught my eye.

The Lightning Process is the woo-woo pseudo-nonsense that trains people to reject their illness and engage in positive affirmations, among other strategies. Professor Esther Crawley, the University of Bristol pediatrician whose work is fraught with ethical and methodological missteps, published a study last year that purported to document its effectiveness in treating kids. But the study violated core scientific principles, as I have documented on Virology Blog and in correspondence with the editor of the journal that published it, Archives of Disease in Childhood.

The editor has failed to take any action. So has BMJ, which publishes Archives of Disease in Childhood. This dereliction of editorial responsibility is certainly disturbing, since the facts of the case are incontrovertible: 56 out of the 100 trial participants were recruited before registration as part of a feasibility study, and the outcome measures for the final paper were swapped based on the results of these early enrolled patients. These maneuvers are clearly guaranteed to induce bias and are unacceptable from any scientific perspective.

Here€™s an e-mail I sent last Thursday to two key members of the NICE guidance executive, Sir Andrew Dillon and Professor Mark Baker. I also cc€™d several stakeholders as well as Fiona Godlee, the editor-in-chief of The BMJ and editorial director of BMJ Company.

According to the company’s site, Dr. Godlee is “responsible for ensuring the editorial integrity of all or [sic] our products and services.” Considering Dr. Godlee’s continuing silence on the documented ethical violations in the Lightning Process study, she does not appear to be taking these responsibilities too seriously.

**********

Dear Sir Andrew and Professor Baker,

I am a research fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last couple of years, I have been investigating the PACE trial and the associated body of research into GET and CBT as treatments for the illness variously known as CFS, ME, CFS/ME, and ME/CFS (we can agree to ignore the name SEID). Along with many others, I have previously alerted NICE to the serious problems with the claims that these treatments are effective.

I now want to express concern that the draft scoping report includes mention of the Lightning Process as a possible treatment. I assume this suggestion is based on the study that appeared last fall in Archives of Disease in Childhood, one of the journals from the BMJ Company. As commenters on a patient forum noted, and as I documented on Virology Blog, more than half of the study participants were recruited before the trial registration, as part of an earlier feasibility study. Publication of such a study represents a clear violation of BMJ€™s own policies as well as those promulgated by the Committee on Publication Ethics.

Here is my post on the problems with the Lightning Process study: https://www.virology.ws/2017/12/13/trial-by-error-the-smile-trials-undisclosed-outcomes/

The Lightning Process feasibility study began recruitment in September, 2010. The full study was registered, as a prospective trial–in July, 2012. Yet of the 100 participants included in the full trial, 56 were recruited before registration as participants in the feasibility study, according to the available documentation. Thus, this was not really a prospective trial.

In addition, when the investigators applied to the local research ethics committee to extend the feasibility study into the full trial, they simultaneously requested approval to swap primary and secondary outcomes based on the results from the feasibility study participants. Swapping outcomes based on early results of trial participants is obviously a recipe for bias, so it is perplexing that the research ethics committee approved the investigators€™ application.

The primary outcome for the feasibility study, school attendance at six months, became a secondary outcome in the full trial. Meanwhile, a secondary outcome in the feasibility trial, self-reported physical function, became the primary outcome in the full trial. The investigators presented positive results for the new subjective primary outcome of physical function. They reported null results for school attendance at six months, the initial primary outcome but now a secondary one. Not surprisingly, media reports highlighted the positive results for the revised primary outcome and ignored the null findings for the initial primary outcome.

Moreover, the published paper in Archives of Disease in Childhood did not mention that more than half of the participants were recruited before trial registration as part of the feasibility study. Nor did the paper disclose the salient detail that primary and secondary outcomes were swapped midway through the trial, based on data obtained from these feasibility study participants.

The Committee on Publication Ethics has a longstanding policy that papers in which participants were recruited before trial registration should not be published. BMJ also has such a policy. It is indisputable that Archives of Disease in Childhood violated COPE and BMJ policy in publishing the Lightning Process study.

On January 30th, 21 experts sent a letter to Archives of Disease in Childhood outlining these concerns. The journal€™s editor promised to review the matter and respond. It has been more than three months, and no response has been forthcoming. It should have taken editors no more than a few minutes to check the documentation and recognize that the paper did in fact violate the journal€™s policy. The failure of the editor or anyone else at BMJ to respond to these issues in a timely manner is troubling.

Here is the letter to Archives of Disease in Childhood: https://www.virology.ws/2018/01/30/trial-by-error-a-letter-to-archives-of-disease-in-childhood/

I have cc€™d some of the NICE guidance stakeholders on this e-mail, as well as Fiona Godlee, editor in chief of The BMJ and editorial director of BMJ Company. Since Archives of Disease in Childhood has remained silent on the matter, perhaps Dr. Godlee can provide an acceptable explanation for why this study was published even though it violates BMJ€™s policy about trials in which participants were recruited before registration, not to mention trials in which primary and secondary outcomes were swapped half-way through based on the results of more than half of the participants.

In any event, unless Dr. Godlee can shed some light on the perplexing anomalies of the Archives of Disease in Childhood paper, consideration of the Lightning Process as an intervention should not be part of the NICE guidance review going forward.

Thank you for your consideration of this matter.

Best–David

David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley