By David Tuller, DrPH
*This is a crowdfunding month at UC Berkeley for my Trial by Error project. If you appreciate my work and would like to help support it, here’s the link for this November’s campaign.
UPDATE: Cochrane has rejected Science For ME’s complaint, as discussed yesterday on a thread on the forum at around the time I posted this blog.
Correction: I have removed a sentence stating that Hilda Bastian chose the members of the author team and the Independent Advisory Group herself. These decisions were made collectively and/or in consultation with others at Cochrane. I apologize for the error.
**********
I recently posted about Cochrane’s unsatisfactory responses to a request from the Science for ME forum that the organization withdraw “Exercise therapy for chronic fatigue syndrome.” a seriously flawed and unreliable 2019 review. The forum also requested an update on Cochrane’s delayed efforts to produce a new ME/CFS exercise review designed to replace this 2019 document; the latter, unfortunately, remains on Cochrane’s site and continues to get cited by promotors of the GET/CBT treatment paradigm. The forum found the organization’s responses inadequate and has filed a formal complaint with Cochrane about the matter.
In the meantime, after more than two years of silence about what has—or has not–been happening with the planned new review, Cochrane last week finally offered a glimpse into developments. The information arrived on November 10th in the form of a report from Hilda Bastian, a long-time Cochrane associate and the designated head of a team, called the Independent Advisory Group (IAG), that is charged with overseeing the development of the new review. After promising regular progress reports, she and Cochrane stopped communicating publicly about the guideline development process; until last week, Bastian’s most recent update was in July, 2021. Given this information void, many patients felt dismayed, confused, depressed, and angered. And understandably so.
Bastian started the update with some terrible personal news, then touched on other factors that contributed to delays in communicating with the public as well as to an “unplanned hiatus” of the IAG’s work. As far as I’m aware, this was the first public acknowledgement of such a hiatus. As she wrote:
“This report is long overdue, and so comes with deep apologies for my long silence. I was out of action for some time after the sudden death of my son late in 2021. That coincided with the upheaval for this project caused by the organisational changes and complaints discussed in this report. The combination of extended uncertainties and confidentiality while some processes were underway made meaningful reporting impossible. Those uncertainties included how long the unplanned hiatus for the IAG might last. The authors’ work on the draft protocol continued, however.”
Besides Bastian’s family tragedy, a major cause of the delay turned out to be objections to the entire project from interests presumably aligned with the GET/CBT ideological brigades—those with the most to lose from a new review. As Bastian explained:
“The reasons [for the delay] included the lengthy and confidential process following a detailed complaint that called for the process to be discontinued, and for the review to remain in its current form. The basis for incorporating an IAG into the editorial processes of a Cochrane review was challenged, as well as the need for an update, and the proposed editorial process. The complaint also argued that too many of the people who were authors of the review or members of the IAG, including me, were too biased in favour of criticisms of the current Cochrane review.”
And further: “Multiple committees within Cochrane’s organisation considered this complaint consecutively, which took around a year. The complaint was not upheld after thorough consideration by the Conflict of Interest Panel, a subgroup of the Editorial Board, and finally Cochrane’s Governing Board.
So let me understand this. After promising regular updates regarding a project vital to the ME/CFS patient community, Cochrane undertook a lengthy internal investigation without making any public communication, leaving everyone in the dark for an extended period of time. Was there really no way for the organization to disclose the existence of an obstacle that was delaying matters while preserving whatever confidentiality was required? I am perplexed that anyone would determine that the only or least the best way to handle this difficult situation was to ghost the entire patient community for more than two years. Beyond being unacceptable, this strategy was certainly not the way to build trust and confidence in the process going forward.
On the positive side (sort of): Bastian mentioned a proposal that could help ameliorate some of the possible damage from the decision to not withdraw the review. The plan is for the IAG to create a better editorial note for the review because, as she wrote, “some uses of the Cochrane review do not reflect critical nuance and limitations in its conclusions, and the goal of a note would be to try to reduce such misinterpretation.” (Of course, such an editorial note would not be necessary at all if Cochrane removed the review in the first place.)
Along with posting her report, Bastian opened a blog to serve as a communications platform on which she would respond to comments. That has led to a lively debate—with lots of sharp comments and generally patient but not always satisfactory responses. I don’t envy Bastian’s task here. Cochrane seems to have placed her in the uncomfortable position of having to field continuing criticisms of the 2019 review and to defend the hard-to-defend decision not to withdraw it.
Whatever editorial note has been or will be slapped on the 2019 review to alert readers to the problematic nature of its findings, it should not have been published in the first place; once published, it should have been withdrawn. Bastian herself had nothing to do with the previous version and the decision-making around it. Nonetheless, because Cochrane handled the whole matter so badly and remains recalcitrant in its refusal to take further action, she is bearing the current brunt of complaints about what has and hasn’t happened.
But here’s the bottom line: As an institution, Cochrane seems unable to come to grips with the reality that unblinded trials relying solely on subjective outcomes are fraught with bias and do not provide reliable evidence for therapeutic interventions. This is especially so when the subjective findings are not only modest but are contradicted by poor or null results for objective outcomes in the same body of research—as is the case here. So passing judgement on these studies should not be a hard call. But apparently it is, at least for Cochrane.
That obviously raises some concerns about what kind of review could emerge from this ongoing process. We’ll have to see what happens going forward. But both the writing team and the IAG have some smart people on them, so there’s reason for at least measured optimism that the final product will be acceptable—notwithstanding Cochrane’s questionable behavior with regards to the 2019 version.
David – Thank you for this clear explanation of what’s been going on. To my mind, transparency is key if public-serving organizations want to gain or maintain public trust and a good reputation.
I find it very puzzling not only that the investigation of a complaint about Cochrane initiating a new ME/CFS Exercise Review and the substance of the complaint needed to be confidential, but also the fact that the IAG heading up public consultation needed to be put in limbo for some two years, with further the very existence of the complaint and how it was being dealt with being subject by Cochrane to a gag order.
Cochrane when initiating this new exercise review procedure asserted it would be an exciting new approach to consumer consultation. Then for over two years imposed secrecy with no meaningful response to any communication.
Whilst NICE in the UK was rewriting its own ME/CFS guidelines an anonymous senior member of one of the UK medical Royal Colleges (revealed by a freedom of information request) sought to pressure NICE’s management to an off the record rewrite of their evidence review to support the continued use of exercise based intervention. Various Royal Colleges then attempted to blackmail NICE into stopping publishing those guidelines saying their members would boycott the new guidelines. This behind the scenes lobbying to effectively subvert the agreed evidence review ultimately failed with NICE, whose guidelines rejected the use of exercise as a curative treatment for ME. NICE held a roundtable meeting for stakeholders where the objecting Royal Colleges failed to present any substantive objections and ultimately agreed to the guidelines’ delayed publication going ahead in late 2021, about the same time that Cochrane suspended their IAG and put a blackout on any information relating to what they were doing.
Since then various senior researchers and clinicians have sought to undermine the 2021 NICE guidelines by publishing what they claim to be research pieces attacking NICE and defending the use of exercise therapy for ME/CFS. However these opinion pieces boil down to asserting personal views justified only by the status of the people who hold them, what Brian Hughes describes as eminence based medicine. Similar pieces were written to defend the indiscriminate use of blood letting in the nineteen century.
This secrecy on the part of a charitable organisation, funded in part by various governments, and supposedly devoted to promoting the very best of scientifically evaluated medicine, is profoundly worrying. It may have been coincidence, but no public response about what was happening came from Cochrane until Science for ME sent them a public letter, supported internationally by some 60 ME/CFS and Long Covid organisations requesting the retraction of the flawed 2019 Cochrane Exercise Review. Equally worrying now is that Cochrane’s non response to any of the substantive issues raised by S4ME appears to be a policy decision. Cochrane having further declined to consider S4ME’s official complaint mentioned by David above, saying the matters have been fully dealt with, without any indication of how or when.
Though the development of a protocol for publication about how Cochrane will go about gathering evidence for a new exercise review has been recommenced we have no idea how long that will take or even if the full review process ever be completed sufficiently to replace the 2019 Review that Cochrane agreed was flawed when it was first published.
It very much feels that a relatively small group of eminent clinicians, having lost the public debate on the science, are now seeking to continue the argument by back door channels. NICE held firm, but Cochrane is wavering, and, though their new review process has resumed after a two year plus delay, their fatally flawed 2019 Exercise Review remains in the Cochrane Library and continues to be cited to defend the use of exercise based intervention in ME/CFS and Long Covid even though it is counter indicated for anyone experiencing post exertional malaise, the core symptom of ME/CFS and of around 50% of those with Long Covid.
Cochrane fully accepts that unblinded trials relying solely on subjective outcomes are fraught with bias when it comes to pharmacological and surgical interventions. They deliberately apply a double-standard to psychotheraputic interventions because they’d rather promote low quality evidence than upset the status-quo.
I can’t disagree with anything that’s been said here.
Yes, thanks for this very clear explanation albeit I’m not up to speed on all nuances with this. But if there was ever a situation of politics in medicine, this certainly seems to me to be a perfect example of such. Very sad for the entire medical community come to think of it as I would think any medical practitioner would want politics to be as far removed from medical science as is possible.
David wrote, among other things, that ” the IAG have some smart people on them, so there’s reason for at least measured optimism that the final product will be acceptable”.
We can hope.
David also wrote “ … ‘Exercise therapy for chronic fatigue syndrome.’ a seriously flawed and unreliable 2019 review .. should not have been published in the first place” …
I have repeatedly wondered if some of this type of event would be prevented if there was a widespread return to ‘scientific falsification’ (On The Origin Of Another Genetic Constant – 2). Is it gone from our scientific endeavors now?
Shameful! Thank you for continuing to cover this, Dr. Tuller!!
My understanding is that Cochrane was really good when it started. It started as a remedy to the type of junk biased science in its ME Review. Unfortunately, as I understand it, over time Cochrane became “captured” by industry who have used it as just another forum for its mala fide “science.” A Co-Founder, Peter Gotszche, was forced pout over this and 4 of the other 11 Directors quit in protest..