By David Tuller, DrPH
Earlier today, I e-mailed the following letter to the members of the House of Commons Science and Technology Committee, which has been investigating issues of scientific and research integrity. I thought it would be a good idea to make sure they knew that this quality appeared to be lacking in some studies in the ME/CFS domain. (If the letter sounds familiar, that’s because it is pulled together from previous versions I’ve sent to others.)
I cc’d both Richard Horton, editor of The Lancet, and Fiona Godlee, editorial director of BMJ.
The subject line: “Concerns about lack of research integrity in Lancet, BMJ papers on ME/CFS”
**********
Dear Rt Hon Norman Lamb MP (and other members of the House of Commons Science and Technology Committee)–
My name is David Tuller. I am a senior fellow in public health and journalism at the Center for Global Public Health at the University of California, Berkeley. For the last three years, I have been investigating the research underlying treatments for the illness variously known as chronic fatigue syndrome, myalgic encephalomyelitis, CFS/ME, and ME/CFS. I am contacting you because of your investigation into scientific integrity, a trait that appears to have been in remarkably short supply in this domain of inquiry.
Through my series of reports (Trial By Error) on a well-regarded science site (Virology Blog), I have helped to highlight serious problems with the PACE trial–a topic addressed in a recent Westminster Hall debate on the illness. Just this week, Virology Blog sent, and posted online–an open letter to The Lancet and editor Richard Horton about the unacceptable methodological lapses in PACE. The letter was signed by more than 100 scientists, academics, clinicians and other experts, 10 MPs, and more than 60 patient/advocacy groups from around the world.
Here is a link to the open letter: https://www.virology.ws/2018/07/10/trial-by-error-yet-another-appeal-to-the-lancet-with-more-on-board/
Here is a link to my 15,000-word investigation of PACE, published in October, 2015: https://www.virology.ws/2015/10/21/trial-by-error-i/
Here is a link to an updated list of all Virology Blog’s posts about PACE and related ME/CFS issues: https://www.virology.ws/mecfs/
I have cc’d Dr Horton on this e-mail. Perhaps he can explain why his journal published a study in which 13 % of the participants were identified simultaneously as being disabled and within normal range for physical function on the same measure, and why the published paper does not mention that salient fact. Perhaps he can also explain why the investigators did not tell participants about their close ties to disability insurance companies, even though they promised in their trial protocol to disclose any possible conflicts of interest as part of the process of obtaining informed consent.
When it comes to poor research in the ME/CFS domain, however, it would be unfair to criticize The Lancet alone. BMJ journals have also published questionable studies. Given your ongoing inquiry into these matters, I felt it was appropriate to bring your attention to two such problematic studies, one in BMJ Open and one in Archives of Disease in Childhood.
The BMJ Open paper, called Unidentified Chronic Fatigue Syndrome/myalgic encephalomyelitis (CFS/ME) is a major cause of school absence: surveillance outcomes from school-based clinics, was published in 2011. The Archives of Disease in Childhood paper, called Clinical and cost-effectiveness of the Lightning Process in addition to specialist medical care for paediatric chronic fatigue syndrome: randomised controlled trial, was published last year.
Both studies violated core methodological and/or ethical principles. Yet because they remain in the literature, they are in a position to impact public health policies and the medical treatments available to kids. Although BMJ editors have been informed of the issues, they have so far failed to take the appropriate and necessary steps. This failure is arguably placing children at risk and raises questions not just about BMJ’s poor decision-making in the prepublication period but about lack of accountability for correcting the record when irrefutable errors are identified.
In the BMJ Open study, the investigators sought to test the hypothesis that children with chronic fatigue syndrome could be identified through school absence records. But they exempted the study from ethical review on the false grounds that it was service evaluation and not research. The study relied on primary data collected directly from identified participants, not on anonymous secondary data, and included a hypothesis and generalizable conclusions. According to UK guidelines, a study with these features is not service evaluation but research, and ethical review is required.
A peer reviewer for BMJ Open raised tough questions about the study’s claim that it was not research but service evaluation. Although the investigators failed to provide an adequate response for the lack of ethical review, BMJ Open overlooked the issue and accepted the paper anyway. Ironically, the journal published it under the heading of research but has since defended the decision to categorize it as service evaluation.
In the Archives of Disease in Childhood study of the Lightning Process, more than half the participants were recruited as part of a feasibility study starting almost two years before trial registration. Even as these feasibility study participants were folded into the full trial, primary and secondary outcome measures were swapped based on their results–an obvious recipe for biasing the findings. The paper in Archives of Disease in Childhood reported positive results but did not disclose these relevant details, a disturbing omission.
To protect against bias, BMJ and other leading journals all have a longstanding policy against publishing trials in which participants were recruited before registration. The Lightning Process study clearly breached this policy. And yet Archives of Disease in Medicine [sic] continues to provide practitioners of the Lightning Process with bragging rights that their pseudo-scientific program is effective in treating children with this illness. The news coverage of the reported results has likely driven more desperate parents to spend their hard-earned money on the Lightning Process. Last month, Archives appended an opaque “editor’s note” to the article; the note indicated that the journal was looking into concerns about some irregularities, but did not provide details or answers.
You can read more about the documented problems with these studies, and letters of concern sent to the journal editors by more than a dozen scientists, clinicians and other experts, in the following Virology Blog posts:
Trial By Error: No Ethical Review of Crawley School Absence Study
Trial By Error: A Letter to BMJ Open
Trial By Error: The SMILE Trial’s Undisclosed Outcome-Swapping
Trial By Error: A Letter to Archives of Disease in Childhood
I have cc’d Fiona Godlee, editor-in-chief of The BMJ and editorial director of BMJ Company, on this e-mail. Perhaps she can explain why journals under her purview have not yet publicly acknowledged the obvious: The BMJ Open study was not service evaluation, and the Archives study violated BMJ policy on prospective trial registration. Neither should have been accepted for publication in the first place.
Does Dr Godlee believe these incidents suggest a need for greater scrutiny of BMJ’s editorial and peer-review processes, at least when it comes to this domain of inquiry? Does she have suggestions for what recourse should be available when journals fail in their responsibility to monitor themselves, with potentially negative public health and medical consequences?
I am raising these issues of scientific integrity with you now because this BMJ research is at risk of being given credence in deliberations on public policy involving children. That includes such initiatives as the current NICE effort to develop a new guidance for ME/CFS or the ongoing roll-out of the problematic cost-cutting program called Improving Access to Psychological Therapies. Like PACE, these two flawed studies have no place being cited in support of any public health initiatives, clinical guidelines, or recommendations on medical care. But because of apparent editorial deficiencies at the journals in question, the papers somehow passed peer review, were accepted for publication, and can be cited as authoritative sources.
Thank you for your attention to this important matter. I would be happy to answer any questions you might have, whether about these two papers, the PACE trial, or other studies from this subpar body of research.
Best–David
David Tuller, DrPH
Senior Fellow in Public Health and Journalism
Center for Global Public Health
School of Public Health
University of California, Berkeley
Comments are closed.