Trial By Error: CODES Trial Commentary Promotes ‘Eminence-Based Medicine’

By David Tuller, DrPH

By all accounts, the recently published CODES trial was the most authoritative study to date of whether cognitive behavior therapy (CBT) was an effective treatment for so-called dissociative seizures–a point confirmed in a commentary accompanying the paper in Lancet Psychiatry. Unfortunately, the CODES investigators and the commentary author seem to interpret the null results for the primary outcome as a call to find other assessment measures–not as a reason to seriously question the therapeutic intervention and its theoretical foundation.

The commentary author, Dr David Perez, is a psychiatrist and neurologist at Massachusetts General Hospital in Boston and an expert on functional neurological disorders (FND)–a more recent name for what many still call conversion disorders. The FND category includes dissociative seizures, more commonly known as psychogenic non-epileptic seizures (PNES). His commentary makes a nod in the right direction, asking whether the null findings for the primary outcome of seizure frequency per month mean that CBT might not be effective for this medical condition.

Notably, however, he does not point out that this wasn’t even a close call. The CODES results demonstrated a tendency, albeit not reaching statistical significance, toward greater seizure reduction in the group that did not receive the intervention, as I pointed out in a recent post.

Despite the null results for the primary outcome in this major trial, Dr Perez answers his question about whether CBT is effective in the affirmative, apparently based on first-hand observation of “some patients” and “certain individuals.” He writes this: “In my opinion, CBT remains an effective treatment for dissociative seizures. I have witnessed how some patients with dissociative seizures can benefit greatly from a CBT approach that equips the patient with new psychotherapeutic tools, which in certain individuals can lead to robust and sustained clinical improvement.”

The Irish psychologist Brian Hughes has used the phrase “eminence-based medicine” to capture the type of logic on display here. Personal witness should be valued and appreciated, in medicine and in life, but it is not a convincing counterweight to the findings of the largest study to investigate CBT for dissociative seizures. Dr Perez mentions that CODES reported benefits from the intervention on nine of 16 secondary outcomes, but he omits an important detail. Only five of these outcomes were statistically significant after correcting for multiple analyses–the standard statistical procedure when so many outcomes are included in a study, as I also explained in the earlier post.

In contrast to the commentary’s claims, the findings of benefit from the CBT delivered in this study are weak. Even those findings cannot be taken at face value, since the study is an open-label or unblinded trial relying on self-reported findings–an experimental design known to be highly vulnerable to bias. And “self-reported” in this case includes the number of seizures, as recorded by participants in seizure diaries. Given the study characteristics, it wouldn’t be surprising if those who knew they were receiving a treatment expected to reduce seizures ended up reporting their seizures differently than those not receiving the treatment.

It is obviously a challenge for investigators when blinding treatment allocation is impossible, as is often the case with non-pharmacological interventions like this one. But research challenges do not justify making expansive claims not backed by the data. In the previous post, I included a blunt assessment of the CODES findings from my colleague Philip Stark, a professor of statistics at Berkeley. Professor Stark’s assessment bears repeating:

“The trial did not support the primary clinical outcome, only secondary outcomes that involve subjective ratings by the subjects and their physicians, who knew their treatment status. This is a situation in which the placebo effect is especially likely to be confounded with treatment efficacy. The design of the trial evidently made no attempt to reduce confounding from the placebo effect. As a result, it is not clear whether CBT per se is responsible for any of the observed improvements in secondary outcomes.”

**********

Is Seizure Frequency the Best Primary Outcome?

After Dr Perez’ quick dismissal of any concerns about CBT effectiveness, he pivots in his commentary to what he views as a key lesson of the research: Seizure frequency, he suggests, is not the best primary outcome anyway. In this way, he sidesteps the implications of the undesirable findings by engaging in a sort of post-hoc outcome-swapping–denigrating the primary outcome as inadequate while elevating the importance of the secondary ones that demonstrated apparent benefit.

But this flip-flop raises some questions. If seizure frequency is not the proper choice for primary outcome, why was this not clear to the investigators after the rather equivocal results from their pilot study–and before they launched a huge, expensive trial? What is the evidence that seizure frequency is the wrong primary outcome measure, beyond results that disappointed the investigators and other CBT advocates?

Dr Perez cites a 2017 study–also cited by the CODES investigators–to support his statement that “associations between dissociative seizure frequency and health-related quality of life are complex and multifactorial, and psychosocial and psychiatric factors might relate more closely to health-related quality of life than seizure frequency itself.” As I explained in the previous post, to draw such a conclusion from the associations documented in this cross-sectional study is unwarranted.

Moreover, if outcomes assessing health-related quality of life are the main metrics of interest, as Dr Perez indicates, then the evidence from CODES itself is not promising. Of the nine outcomes that were statistically significant before correction for multiple analyses, only one was on an instrument measuring health-related quality of life. And of the seven outcomes that did not meet statistical significance, two were separate physical and mental scales on a different health-related quality of life instrument.

After the CODES data were corrected for multiple analyses, none of these three health-related quality of life outcomes were found to have yielded statistically significant results. So it would be a big and questionable stretch for anyone to claim that the CBT intervention road-tested in CODES led to improvements specifically in health-related quality of life, no matter what anecdotal examples Dr Perez or others can cite from their clinical work.

**********

Failure to Grapple with a Core Question

Dr Perez and the CODES investigators fail to grapple adequately with a central question: Was the hypothesis behind the CBT intervention wrong? As described in the 2010 pilot study, the approach was grounded in the notion that these seizures were “psychogenic”–the study used the common PNES label–and “lacking organic etiology.” The expectation was that cognitive restructuring plus seizure-reduction strategies would address the causes or triggers of these seizures that lacked organic etiology, leading to their reduction or elimination. That didn’t happen.

The CODES investigators and Dr Perez appear to have avoided repeating similar sorts of declarations about non-organic etiology. But is that still the operative hypothesis behind the CBT intervention, which Dr Perez says is effective? If the etiological hypothesis remains the same, why didn’t the intervention work as expected? And if the etiological hypothesis has evolved as the understanding of dissociative seizures has become more nuanced, why hasn’t the intervention also evolved to accommodate that? (If such changes in the intervention occurred, CODES does not highlight them.) What is the current rationale for a CBT intervention that was designed to treat what was categorically identified as a non-organic condition?

Dr Perez suggests that perhaps, given the heterogeneity of this patient population, it would be better not to lump everyone into a single category. That is undoubtedly an excellent idea. Perhaps if experts like the CODES investigators, decision-makers at UK funding agencies, and others had not been so invested on promoting CBT as a one-size-fits-all treatment for this obviously complex phenomenon, research into sub-categories would be more advanced than it is.

In his commentary, Dr Perez focuses on subgroups of dissociative disorder patients who have been diagnosed with co-morbid psychiatric disorders. And he recommends other unproven psychotherapies as options for treatment–even though CODES has just documented that CBT does nothing more than offer adjunct support that might help some people feel better. Apart from mentioning those who experience dissociative seizures after head trauma, he does not include any suggestion that some proportion of these patients might be suffering from undiagnosed but actual neurological or other diseases characterized by pathophysiological processes.

I have no doubt that psychiatrists and neurologists care deeply about their patients and are often desperate to help those with complaints that seem to resist resolution, including dissociative seizures. Dr Perez comes across as a thoughtful clinician, and it is not surprising that some dissociative seizure patients in his care would experience psychosocial relief after a course of CBT. Perhaps some of his patients have even experienced a remission of symptoms, for whatever reason.

But post-CODES, it is disingenuous to argue that CBT constitutes an evidence-based treatment for dissociative seizures. According to the best and most definitive data now available, this intervention does nothing to impact the primary outcome that experts have spent more than a decade hailing as the most important measure of treatment success.

**********

Lancet Psychiatry Rejects a Letter about the Commentary

Joan Crawford, a UK psychologist, wrote to Lancet Psychiatry recently concerning the Perez commentary’s apparent reliance on what Irish psychologist Brian Hughes has called “eminence-based medicine.” Here’s part of her letter:

“Bias is evident in Perez’s commentary. For example, ‘In my opinion, CBT remains an effective treatment for dissociative seizures.’ The CODES trial does not support his claim. He wishes this to be the case; but it is not so. Humans are largely loss averse. It can be hard to step back from a strongly invested belief even in the presence of contradictory evidence. His desire to continue to believe in the trial’s success is a powerful demonstration of how bias works.

Perez cites evidence from his clinical practice. I do not doubt his sincere belief of robust improvement via CBT. However, we are unable to evaluate its objectiveness. Whether he is confusing “feeling better” (adapting to living well despite ongoing symptoms) with˜being better” (cured/recovered) is impossible to ascertain. Patients heartily desire the latter. Honesty about the limitations of current practice is needed.”

I concur with these points. Unfortunately, Lancet Psychiatry rejected the letter, pleading an excess of coronavirus news. Per a deputy editor: “Normally, we would welcome a debate spurred by a Comment but I am afraid that COVID-19 is dominating our Correspondence sections across all the Lancet journals and we have been asked to cut everything else back. So on this occasion, we have decided not to publish your letter.”

An appeal of this decision was also rejected. It is unfortunate that Lancet Psychiatry decided to publish a commentary with questionable assertions and then refuse to post a cogent letter about its inadequacies. I obviously don’t doubt the onslaught of coronavirus-related correspondence, but if the journal was not interested in fostering debate on this issue then perhaps it should have reconsidered before publishing the article at this time.

Hopefully, Lancet Psychiatry will deign to host a robust discussion on this matter in the future. But remember, this is the same journal that published the PACE long-term follow-up in 2015, which made laughable claims of success for CBT and GET even though there were no differences between the groups at the end. So don’t expect much.