Trial By Error: Cochrane’s Report on Courtney’s Complaint

By David Tuller, DrPH

Update:

When I posted earlier today, I also meant to update what was happening with the exercise review. On March 8, Cochrane announced, through a notice appended to the review, that the authors would submit another revision in May. (Cochrane has already rejected a previous revision submitted last fall.) That is also the month when Dr Tovey leaves his position as Cochrane’s as editor-in-chiefl it is unclear what that coincidence portends, if anything. Presumably the final decision on the exercise review will rest with Dr Tovey’s successor.

Here is the notice appended to the review:

‘Cochrane’s editors and the review author team have jointly agreed that there will be a further period up to the end of May 2019, in which time the author team will amend the review to address changes aimed at improving the quality of reporting of the review and ensuring that the conclusions are fully defensible and valid to inform health care decision making. The changes will also address concerns raised in feedback since the Robert Courtney complaint. The amendment will not include a full update, but a decision about this will made subsequently.’

As far as Cochrane’s report on Courtney’s complaint…In Norway, people can access documents that public agencies list on this site. The Cochrane report was included in correspondence between the Norwegian Institute of Public Health and Cochrane, a record of which can be found here.

**********

Last fall, the news media reported that Cochrane was planning to temporarily withdraw a 2014 review of exercise therapies for the illness it has long called chronic fatigue syndrome. Those reports appeared to cast blame for the brouhaha on irrational patients, suggesting that Cochrane was succumbing to anti-scientific zealots.

The news coverage generally failed to discuss the actual methodological issues of concern. In response, Virology Blog published an open letter of support for Cochrane’s decision, signed by more than 40 scientists, academics and other experts.

First, to get one point out of the way: The studies included in the exercise review were open-label trials relying largely on subjective outcomes. As has been stated umpteen times during this entire debate, this study design is highly prone to bias and has been abandoned in other areas of medicine. That means the review itself is a mass of unknown amounts of bias, and the review’s conclusions are essentially uninterpretable. Yet Cochrane does not appear at this point to agree that such studies should not be used to determine public health policy; hopefully the organization will come around to this necessary perspective in the near future.

Putting that key issue aside for now, Cochrane’s decision about the exercise review was clearly based on scientific considerations. Two patients, Robert Courtney and Tom Kindlon, had written extensive critiques of the 2014 exercise review. The review’s lead author, Norwegian researcher Lillebeth Larun, failed to provide adequate responses, so Courtney filed a formal complaint with Cochrane. Cochrane assessed that complaint and found merit in Courtney’s concerns. Unfortunately, Courtney died before learning that Cochrane had upheld his complaint.

I now have a copy of Cochrane’s report about Courtney’s complaint. It is clear from the analysis that Cochrane’s concerns involved the scientific accuracy and integrity of the exercise review. Contrary to the claims of the exercise review’s defenders, Cochrane was not responding to a hysterical mob of patients but to the cogent and well-reasoned argumentation of one patient in particular.

Courtney was correct in determining that the review’s methodology allowed the authors to maximize their ability to report benefits from exercise therapy. Were the authors to adhere to these criticisms in revising the review, they would have to significantly downgrade their positive conclusions. This is clear from Cochrane’s inclusion in its report of examples of paragraphs edited in ways that temper the expressed enthusiasm for exercise therapy.

While the Cochrane report spells out many of the issues, it unfortunately does not include discussion of the authors’ decision to omit objective outcomes from the review. In responding to this criticism, Larun justified the omission by noting that these outcomes were not included in the review protocol. This is essentially a non-answer; it just raises the question of why Larun and her colleagues decided to exclude these data at that earlier stage, and why reviewers of the protocol allowed this unacceptable maneuver.

Since objective outcomes have not matched the reports of success on subjective measures, including these results would obviously have impacted the review’s findings. You don’t need to be a genius to figure out that, by ignoring data that contradicted their desired outcomes, Larun and her colleagues heavily biased their review of exercise therapies toward the positive.

Here’s the report: Cochrane Report on Courtney Complaint