Trial By Error: Some Thoughts on Ten Years of Trial By Error

By David Tuller, DrPH

Ten years ago this month, I launched Trial By Error with a 15,000-word investigation of the misbegotten and fraudulent PACE trial, which purported to prove that graded exercise therapy (GET) and cognitive behavior therapy (CBT) could cure what they then called chronic fatigue syndrome. And what an amazing ride it’s been for me—difficult and challenging at times, but always fascinating, rewarding and engaging, both intellectually and personally.

I have learned so much, have heard so many heartbreaking stories, and have met–whether in person or online—so many smart, funny, passionate and courageous people. I have also been deeply disturbed by the rampant and flagrant corruption in the editorial and peer-review practices of major medical journals, including those in the Lancet and BMJ stables. Not only have these august publications routinely failed to catch egregious methodological missteps, they have also routinely failed to take appropriate steps to fix problems once they’ve been pointed out. In many instances, their response to legitimate criticism can only be described as a blatant “fuck you.”

Back when, I never aspired to be the “chronic fatigue syndrome reporter” (or the “myalgic encephalomyelitis reporter”). I mean, who would have? But that’s what happened, and I am grateful that it did. Although I didn’t plan on spending an entire decade—basically, my 60s—on this project, it is hard now for me to imagine a more fulfilling or better use of my time.

The Lancet published the initial PACE results in 2011. Four years later, my initial PACE investigation appeared over the course of three days on Virology Blog, a popular science site hosted by my friend and colleague Vincent Racaniello, a microbiology professor at Columbia University. Given the length, the series was likely, for many patients, impossible to read without triggering major bouts of post-exertional malaise (PEM).

(As always, I need to point out that patients were aware of the trial’s fatal flaws long before I got involved; my work piggy-backed on their incisive and spot-on analyses of the research. I obviously owe enormous thanks to Professor Racaniello for supporting my project and allowing me to regularly hijack his site to disseminate my findings. I’m also grateful to Valerie Eliot Smith, a lawyer and longtime patient, and her husband, Robin Callendar; in addition to offering invaluable advice on legal matters, they suggested the name Trial By Error, which has stood the test of time.)

The publication of Trial By Error was very timely. A week later, Lancet Psychiatry published the PACE trial’s unimpressive long-term follow-up results, assessed on average 2.5 years after participants entered the trial. for the PACE trial. (The initial results were from the 12-month assessments.) This publication was another anti-scientific production from the high-powered triumvirate of lead authors—psychiatrists Michael Sharpe and Peter White, professors at, respectively, Oxford University and Queen Mary University of London, and Trudie Chalder, King’s College London’s factually and mathematically challenged professor of cognitive behavior therapy.

What was wrong with the follow-up study? Plenty. The authors chose not to focus on differences in long-term outcomes between the GET and CBT intervention groups and the comparison groups, since there were no such differences. Instead, they highlighted as their main finding that the intervention groups had maintained the (very modest) improvements claimed in the initial trial publication in The Lancet.

The fact that in the end the GET and CBT treatments provided no apparent advantage was relegated to a side-point—even though the results of interest in any clinical trial, even in follow-up, are the between-group differences, not whether the active intervention groups maintained their gains. In other words, presenting the follow-up findings as if they demonstrated the purported superiority of CBT and GET was a flagrant violation of scientific standards. It also represented a stark failure of the journal’s peer-review practices.

However, because Trial By Error had appeared the week before, some high-profile publications–such as Science (“Criticism mounts of a long-controversial chronic fatigue [syndrome] study”) and The Guardian (“Chronic fatigue [syndrome] patients criticise study that says exercise can help”)–reported on the Lancet Psychiatry paper and my expose in the same article. When these more balanced and nuanced media accounts appeared, it gave me great pleasure to imagine the dismay and shock they must have triggered in the three prominent academics in charge of PACE.

After all, these privileged and self-important investigators were used to glowing coverage of their work. From my perspective, some of this unwarranted praise could only be attributed to the British tradition of deference to authority—in this case, those who had risen to the level of. “professor.” In American English, “professor” is often used generically to designate anyone teaching students above the high school level. At Berkeley, for example, my students would routinely refer to me as their professor, even though I was officially a “lecturer.”

In the UK, it seemed to me, anyone anointed with an official professorship commanded unquestioning respect—whether deserved or not. In the case of the PACE authors and their colleagues, such respect was clearly not justified. Even so, patients challenging the study findings, no matter how cogent and accurate their arguments, were routinely portrayed as twisted and dangerous terrorists bent on tearing down the august domain of Science. In reality, the PACE authors and their ilk were the deluded batch, promoting bogus recommendations that posed enormous potential harm to patients, given the risk of prolonged bouts of PEM.

That an academic from a world-class research institution in the United States would call the trial “a piece of crap” and publicly rip a print-out of the trial to shreds was apparently outside these professors’ range of experiences and expectations. After the initial publication, QMUL declared that my work was triggering “internet abuse” and “reputational damage”—language that I interpreted as legal threats. Of course, I wasn’t sued then and I haven’t been since, even though British libel laws are far more biased toward the plaintiffs than in the U.S.

**********

Over the years, the PACE authors and their colleagues have made fools of themselves in trying to defend the indefensible. I could fill a book with tales of their stupidity and arrogance, but will just recount a couple of examples here.

Not long after the publication of Trial By Error, Professor Sir Simon Wessely, who had a major role in PACE although was not a listed author, immediately sought to support his colleagues with a misguided essay in The Mental Elf, a prominent site covering mental health issues. In the essay, called “The PACE Trial for Chronic Fatigue Syndrome: choppy seas but a prosperous voyage,” Professor Sir Simon compared the study to an ocean liner that sets out from Southampton to New York. After a few mid-course “corrections,” he noted, the “HMS PACE” arrived at last at its intended destination.

(The essay did not mention me or Trial By Error, although the post was clearly an effort to debunk what I had exposed.)

This was an idiotic and ill-advised analogy. Apart from calling to mind the Titanic, it represented a fundamental misunderstanding of the point of clinical trials—this from someone who fancied himself an expert in research methodology. As my friend and colleague Steve Lubet, a law professor at Northwestern University in Chicago (now emeritus), wrote in an open letter to Professor Sir Simon posted as a blog:

“You compare the PACE Trial to an ocean liner plotting a course from Southampton to New York, and express satisfaction that it made the trip “successfully across the Atlantic,” despite course corrections along the way.  But surely you realize that a randomized controlled study is not supposed to have a fixed destination, but rather should follow wherever the evidence – or the current, to maintain the metaphor — leads. You thus virtually admit that the PACE Trial was always intended to reach a particular result, and that adjustments along the way were necessary to get it there.  Just so.…”

Oops. Busted! In his response to Professor Lubet’s blog post, Professor Sir Simon dribbled out some drivel and nonsense, but the damage was done.

For his part, the hapless Professor Sharpe also tried to challenge Trial By Error’s reporting—to little avail. One of my sources, Columbia University’s biostats professor Bruce Levin, had referred to some of the PACE trial’s methodological missteps as “the height of clinical trial amateurism”—a potent phrase that made me laugh. Poor Professor Sharpe apparently took umbrage—and months after Trial By Error appeared, he e-mailed Professor Levin to ask whether he’d actually described the research in this manner.  

Professor Levin informed him that, yes, he did indeed say what I’d quoted him as saying, and he shared with Professor Sharpe some of the other sharp criticisms he’d made as well. After Professor Levin informed me of their exchange, I e-mailed Professor Sharpe and offered to provide him with contact information for the other academic experts I had quoted. Perhaps, I suggested, he might want to vet all of their quotes in the story and assure himself that they all genuinely viewed his beloved trial as bullshit. Professor Sharpe declined my offer.

That was all quite a while ago now. To the discredit of The Lancet, PACE has not been retracted. It has, however, been discredited—and good riddance. Since those early years, the PACE authors and their colleagues have exhibited increasing desperation to protect their intellectual and academic turf. They continue to pretend their research has been robust and meaningful, and view all their critics as losers, creeps or charlatans. (I think they place me in the latter category, or perhaps even all three.)

Whatever. These academics are classic examples of what are known, in literature, as “unreliable narrators.” Nothing they say or write can be taken at face value. Everything must be presumed to be geared toward protecting their inflated reputations and their privileged status. However, they have lost control of the narrative. They know that, and they are scared. They continue to bleat in protest, but their three-decade hegemonic reign in this domain is over. The world is moving on—yet they refuse to budge.

.