Trial By Error, Continued: The New FITNET Trial for Kids

By David Tuller, DrPH

David Tuller is academic coordinator of the concurrent masters degree program in public health and journalism at the University of California, Berkeley.

The past year has been a disaster for proponents of the PACE trial. They have faced growing international resistance to their exaggerated claims that cognitive behavior therapy and graded exercise therapy are effective treatments for chronic fatigue syndrome, also known as ME/CFS. The recent court-ordered release of key trial data has confirmed what was long self-evident: The PACE authors weakened their outcome criteria mid-stream in ways that allowed them to report dramatically better results for “improvement” (in The Lancet in 2011) and “recovery” (in Psychological Medicine in 2013). In refusing to provide the findings per the original protocol methods, or statistical analyses assessing the impact of the many mid-trial changes, or their actual trial data, they were able to hide their disastrous results for five years.

Yet the PACE authors and their allies continue, astonishingly, to defend the indefensible study, cite its findings approvingly, and push forward with ever more research into behavioral and cognitive interventions. The latest case in point: Esther Crawley, a British pediatrician and a highly controversial figure in the ME/CFS community because of her longtime promotion of the CBT/GET approach. On November 1^st, the Science Media Centre in London held a press briefing to tout Dr. Crawley’s current venture–FITNET-NHS, a one-million-pound study of online CBT that is now recruiting and seeks to enroll more than 700 adolescents.

Dr. Crawley is a professor of child health at the University of Bristol. She is also currently recruiting for the MAGENTA study of graded exercise therapy for children with the illness. She is a lead player in the U.K. CFS/ME Research Collaborative, an umbrella organization that is sponsoring an ambitious Big Data effort called MEGA, now in the planning stages. While patients and advocates are desperate for the kind of top-notch biomedical and genetic research being proposed, many oppose MEGA precisely because of the involvement of Dr. Crawley and Peter White, the lead PACE investigator. (Dr. White is reportedly no longer involved in MEGA; Dr. Crawley still definitely is.)

The rationale for FITNET-NHS is that many ME/CFS patients live too far from specialists to obtain adequate care. Therefore, CBT delivered through an online portal, along with e-mail communication with a therapist, could potentially provide a convenient answer for those in such circumstances. The SMC press briefing generated widespread and enthusiastic news coverage. The BBC’s breathless online report about the “landmark” study noted that the online CBT “successfully treats two-thirds of children with chronic fatigue syndrome.” According to the BBC story, the intervention was designed “to change the way the children think of the disease.”

The BBC story and other news reports did not mention that the PACE trial–a foundational piece of evidence for the claim that changing people’s thoughts about the disease is the best way to treat it–has been publicly exposed as nonsense and is the subject of a roiling worldwide scientific debate. The stories also didn’t mention a more recent paper by the authors of the 2012 study from the Netherlands that was the source of the BBC’s claim of a “two-thirds” success rate.

The 2012 study, a Dutch version of FITNET, was published in The Lancet. (Why is The Lancet always involved?) In a subsequent paper published in Pediatrics in 2013, the Dutch team reported no differences among their trial participants at long-term follow-up. In other words, as with the PACE trial itself, any apparent advantages conferred by the investigators’ preferred treatment disappeared after the study was over. (More on the Dutch study below.)

The SMC, a purportedly neutral arbiter of science, actually functions as a cheerleader for research about cognitive and behavioral treatments for ME/CFS. Simon Wessely, a founder of the CBT/GET treatment paradigm and a close colleague of the PACE authors, is on the SMC’s board of trustees. The journalist who wrote the BBC story, James Gallagher, sits on the SMC’s advisory committee, so the reporting wasn’t exactly conducted at arm’s length. This reportorial conflict-of-interest was not disclosed in the BBC story itself.

(In fact, the Countess of Mar, a member of the House of Lords and a longtime advocate for ME/CFS patients, has filed a formal complaint with the BBC to protest its biased reporting on FITNET-NHS. In her complaint, she noted that “the BBC coverage was so hyperbolic and it afforded the FITNET trial so much publicity that it was clearly organised as a counter-punch to the anti-PACE evidence which is now gaining world-wide attention.”)

As a treatment for chronic fatigue syndrome, cognitive behavior therapy is grounded in an unproven hypothesis. According to the theory, the cause of patients’ continuing symptoms is a vicious downward spiral generated by false illness beliefs, a fear of engaging in activity, and progressive deconditioning. Whatever the initial viral or other illness that might have triggered the syndrome, patients are presumed to be currently free of any organic disease. Changing their beliefs through CBT, per the theory, will help encourage them to increase their levels of activity and resume their normal lives.

Here’s the rationale for the treatment from the PACE study itself: “CBT was done on the basis of the fear avoidance theory of chronic fatigue syndrome. This theory regards chronic fatigue syndrome as being reversible and that cognitive responses (fear of engaging in activity) and behavioural responses (avoidance of activity) are linked and interact with physiological processes to perpetuate fatigue. The aim of treatment was to change the behavioural and cognitive factors assumed to be responsible for perpetuation of the participant’s symptoms and disability. Therapeutic strategies guided participants to address unhelpful cognitions, including fears about symptoms or activity by testing them in behavioural experiments.”

The goal of this specific form of CBT, therefore, is to reverse the “reversible” illness by helping patients abandon their “unhelpful” beliefs of having a medical disease. This is definitely not the goal of CBT when it is used to help people cope with cancer, Parkinson’s, or other illnesses; no one claims those diseases are “reversible.” That the PACE authors, Dr. Crawley, and their Dutch colleagues promote CBT as a curative treatment and not simply a management or adaptive strategy is clear from their insistence on using the word “recovery”–a term that has no well-defined or universally understood meaning when it comes to this illness but has a very clear meaning to the general public.

While PACE so far remains in the literature, the study has been rejected by dozens of leading clinicians and academics, in the U.S. and elsewhere. Last February, an open letter to The Lancet signed by 42 experts and posted on Virology Blog condemned its egregious flaws, noting that they “have no place in published research.” The study has even been presented as a case study of bad science in graduate epidemiology seminars and at major scientific gatherings.

*****

Like the work of the PACE authors, Dr. Crawley’s research is fraught with misrepresentations and methodological problems. Like them, she routinely conflates the common symptom of chronic fatigue with the illness called chronic fatigue syndrome–a serious error with potentially harmful consequences. (I will mostly use chronic fatigue syndrome in describing the research because that is the term they use.)

Dr. Crawley favors subjective over objective outcomes. In PACE, of course, the objective measures–like a walking test, a step-test for fitness, and employment status–all failed to demonstrate “recovery” or reflect the reported improvements in the two primary subjective outcomes of physical function and fatigue. FITNET-NHS doesn’t even bother with such measures. The primary outcome is a self-report questionnaire assessing physical function, and almost all the secondary outcomes are also subjective.

This is particularly troubling because FITNET-NHS, like PACE, is non-blinded; that is, both participants and investigators know which intervention they are receiving. Non-blinded studies with subjective outcomes are notoriously vulnerable to bias–even more when the intervention itself involves telling participants that the treatment will make them better, as is the case with the kind of cognitive behavior therapy provided for ME/CFS patients.

The FITNET-NHS study protocol states that participants will be identified using the guidelines developed by NICE–the U.K.’s National Institute for Health and Care Excellence. The protocol describes the NICE guidelines as requiring three months of fatigue, plus one or more of nine additional symptoms: post-exertional malaise, difficulty sleeping, cognitive dysfunction, muscle and/or joint pain, headaches, painful lymph nodes, general malaise, dizziness and/or nausea, or palpitations. In other words, according to the protocol, post-exertional malaise is not required to participate in FITNET-NHS; it is clearly identified as an optional symptom. (In the U.K., the illness can be diagnosed at three months in children, rather than at six months.)

But the proposal’s claim to be following the NICE guidelines does not appear to be true. In the NICE guidelines, post-exertional malaise is not an optional symptom. It is required, as an essential element of the fatigue itself. (In addition, one or more of ten other symptoms must also be present.) To repeat: post-exertional malaise is required in the NICE guidelines, but is not required in the description of the NICE guidelines provided in the FITNET-NHS protocol.

By making this subtle but significant shift–a sleight-of-guideline, so to speak–Dr. Crawley and her colleagues have quietly transformed their prospective cohort from one in which post-exertional malaise is a cardinal characteristic of the illness to one in which it might or might not be present. And they have done this while still claiming–inaccurately–to follow NICE guidelines. As currently described, however, Dr. Crawley’s new study is NOT a study of chronic fatigue syndrome, as she maintains, but of chronic fatigue.

As a result, the actual study participants, like the PACE cohort, will likely be a heterogeneous grab bag of kids suffering from fatigue for any number of reasons, including depression–a common cause of exhaustion and a condition that often responds to psychotherapeutic interventions like CBT. Some or even many participants–an unknown number–will likely be genuine ME/CFS patients. Yet the results will be applied to ALL adolescents identified as having that illness. Since those who actually have it suffer from the required symptom of post-exertional malaise, an intervention that encourages them to increase their activity levels, like CBT, could potentially cause harm.

(I suppose it’s possible the FITNET-NHS protocol’s inaccurate description of the role of post-exertional malaise in the NICE guidelines was inadvertent, a case of sloppiness. If so, it would be an extraordinary oversight, given the number of people involved in the study and the enormous implications of the switch. It is curious that this obvious and jarring discrepancy between the NICE guidelines and the FITNESS-NHS description of them was not flagged during the review process, since it is easy to check whether the protocol language accurately reflects the recommendations.)

Yet Dr. Crawley is experienced at this blurring of categories–she did the same in a study she co-authored in the journal Pediatrics, in January of this year. In the study, “Chronic Fatigue Syndrome at Age 16 Years,” she and colleagues reported that almost one in fifty adolescents suffered from the illness–an extremely high rate that attracted widespread media attention. The main conclusion was described like this: “CFS affected 1.9% of 16-year-olds in a UK birth cohort and was positively associated with higher family adversity.”

However, the Pediatrics study is unreliable as a measure of “chronic fatigue syndrome.” It is of note that this paper, like the FITNET-NHS protocol, also appears to have inaccurately presented the NICE guidelines. According to the Pediatrics paper, NICE calls for a CFS diagnosis after three months of persistent or recurrent fatigue “that is not the result of ongoing exertion, not substantially alleviated by rest, has resulted in a substantial reduction of activities, and has no known cause.” But this description is incomplete–it omits the NICE requirement that the fatigue must include the specific characteristic of post-exertional malaise in order to render a diagnosis of chronic fatigue syndrome.

In the Pediatrics paper, the determination of illness was based not on clinical examination but on parental reports of children’s unexplained fatigue. In a previous study of 13-year-olds that relied on the same U.K. database, Dr. Crawley and her co-authors referred to the endpointappropriately–as “disabling chronic fatigue.” But in this study, they justified changing the endpoint to “chronic fatigue syndrome” by noting that they cross-referenced the parental reports with children’s self-reports of their own fatigue.

Here’s how they explained this shift in nomenclature: ‘In the earlier study, we were unable to confirm a diagnosis of CFS because we had only parental report of fatigue; hence, chronic disabling fatigue was defined as the study outcome. In the present study, parental and child report of fatigue were combined to identify adolescents with CFS.”

This reasoning is incoherent. A child’s confirmation of a parental report of fatigue cannot be taken to indicate the presence of chronic fatigue syndrome–especially without a clinical examination to rule out other possible conditions. Moreover, neither the parental nor child reports appear to have included information about post-exertional malaise, which is required for a diagnosis of chronic fatigue syndrome–even though the Pediatrics study did not mention this requirement in its description of the NICE guidelines. In fact, the authors provided no evidence or data to support their assumption that a double-report of fatigue equaled a case of chronic fatigue syndrome. (How’d that assumption ever pass peer review?)

Moreover, the study itself acknowledged that, when those found to be suffering from high levels of depression were removed, the prevalence of what the investigators called chronic fatigue syndrome was only 0.6 %. And since depression is likely to be highly correlated with chronic fatigue as well as with family adversity, it is not surprising that the study found the apparent association between family adversity and chronic fatigue syndrome that the investigators highlighted in their conclusion. That misinterpretation of their data has likely lent support to the widespread but inaccurate belief that the illness is largely or even partly psychiatric in nature.

In any event, the figure of 0.6 % should have been identified as the prevalence of “chronic disabling fatigue, not attributable to high levels of depression.” Without any further clinical data, to identify either 1.9 % or 0.6 % as the prevalence of chronic fatigue syndrome was unwarranted and irresponsible. Although the authors cited the lack of clinical diagnosis as a limitation, this acknowledgement does not excuse their interpretive leap. To call this a study of chronic fatigue syndrome is really misleading–a serious over-interpretation of the data.

In subsequent correspondence, three professors of pediatrics–Marvin Medow and Julian Stewart from New York Medical College, and Peter Rowe from Johns Hopkins–scolded the study authors for identifying the participants as having chronic fatigue syndrome rather than chronic fatigue. They cited this misclassification as the likely source of the reported link between chronic fatigue syndrome and family adversity. In particular, they challenged diagnoses made without benefit of clinical evaluations.

“An important component of the diagnosis is a physician’s history and physical examination to exclude conditions that could explain the fatigue, including hypothyroidism, heart disease, cancer, liver failure, covert drug abuse, medication side effects, gastrointestinal/nutritional, infectious and psychiatric conditions,” they wrote. The Pediatrics paper, concluded the three pediatricians, “should be titled ‘Chronic Fatigue but not Chronic Fatigue Syndrome at Age 16 Years.'”

In response, the study authors agreed that clinical diagnoses would be more accurate. But they did not address the critical issue of why they decided that two reports of chronic fatigue could be used to identify chronic fatigue syndrome.

*****

The conflation of chronic fatigue and chronic fatigue syndrome is a huge problem in ME/CFS research. That’s why a major report last year from the National Institutes of Health declared that the case definition used in PACE–which required only six months of unexplained fatigue but no other symptoms–could “impair progress and cause harm,” and should be “retired” from use. But Dr. Crawley and her colleagues do not seem to have gotten the message.

At the SMC press briefing presenting FITNET-NHS, one of the experts appearing with Dr. Crawley was Dr. Stephen Holgate, the leader of the CFS/ME Research Collaborative and a professor of immunopharmacology at the University of Southampton. According to the BBC report, he praised the new trial as “high-quality research.” This endorsement suggests that Dr. Holgate, like Dr. Crawley, does not appreciate the significance of the distinction between the symptom of chronic fatigue and the illness called chronic fatigue syndrome–a troubling blind spot. It also suggests that Dr. Holgate is unaware or unconcerned that the main support for the use of CBT in this illness, the PACE trial, has been discredited.

Also at the SMC briefing was Paul McCrone, a professor of health economics from King’s College London and a PACE co-author. Dr. McCrone is serving as the chair of FITNET-NHS’ independent steering committee–another unsettling sign. As I have documented on Virology Blog, Dr. McCrone made false claims as lead author in a 2012 PLoS One paper–and those false claims allowed the PACE authors to declare that CBT and GET were cost-effective. They have routinely cited this fraudulent finding in promoting the therapies.

Beyond the problem of conflating “chronic fatigue” and “chronic fatigue syndrome,” Dr. Crawley’s reliance on the Dutch trial suggests that this previous FITNET study warrants a closer look–especially since the BBC and other news outlets cited its robust claims of success in extolling the U.K. version.

The approach to CBT in the Dutch FITNET trial reflects that in the U.K. Of the online intervention’s 21 modules, according to the protocol for the Dutch study, fourteen “focus on cognitive behavioural strategies and include instructions and exercises on how to identify, challenge and change cognitive processes that contribute to CFS.” Of course, experts outside the CBT/GET/PACE bubble understand that ME/CFS is a physiological disease and that faulty “cognitive processes” have nothing to do with perpetuating or contributing to it.

The Dutch study found that those assigned to FITNET reported less fatigue, greater physical function, and greater school attendance than those in the comparison group, who received standard treatment–referred to as “usual care.” And using a composite definition of “recovery,” the study reported that 63% of those in the FITNET group–just shy of two-thirds–“recovered” at six months, compared to just eight percent in the comparison group. But this apparent success masks a much more complicated reality and cannot be taken at face value, for multiple reasons.

First, the subsequent 2013 paper from the Dutch team found no differences in “recovery” between participants in the two groups at long-term follow-up (on average, 2.7 years after starting). Those in the comparison group improved after the trial and had caught up to the intervention group, so the online CBT conferred no extended advantages or benefits. The researchers argued that the therapy was nonetheless useful because patients achieved gains more quickly. But they failed to consider another reasonable explanation for their results.

Those in usual care were attending in-person sessions at clinics or doctors’ offices. Depending on how often they went, how far they had to travel and how sick they were, the transportation demands could easily have triggered relapses and harmed their health. In contrast, those in the FITNET group could be treated at home. Perhaps they improved not from the treatment itself but from an unintended side effect–the sedentary nature of the intervention allowed them more time to rest. The investigators did not control for this aspect of the online CBT.

Second, the “recovery” figure in the Dutch FITNET study was a post-hoc calculation, as the authors acknowledged. The protocol for the trial included the outcomes to be measured, of course, but the authors did not identify before the trial what thresholds participants would need to meet to be considered “recovered.” The entire definition was constructed only after they saw the results–and the thresholds they selected were extremely lenient. Even two of the PACE authors, in a Lancet commentary praising the Dutch study, referred to the “recovery” criteria as “liberal” and “not stringent.” (In fact, only 36% “recovered” under a more modest definition of “recovery,” but the FITNET authors tucked this finding into an appendix and Dr. Crawley’s FITNET-NHS protocol didn’t mention it.)

Now, the fact that “recovery” was a post-hoc measure doesn’t mean it isn’t valid. But anyone citing this “recovery” rate should do so with caveats and some measure of caution. Dr. Crawley has exhibited no such reticence; in a recent radio interview, she declared flatly that the Dutch participants had made a “full recovery.” (In the same interview, she called PACE “a great, great study.” Then she completely misrepresented the results of the recent reanalyses of the PACE trial data. So, you know, take her words for what they’re worth.)

Given the hyperbole about “recovery,” the public is understandably likely to assume that Dr. Crawley’s new “landmark” study will result in similar success. A corollary of that assumption is that anyone who opposes the study’s approach, like so many in the patient and advocacy communities, could be accused of acting in ways that harm children by depriving them of needed treatment. This would be an unfair charge, since the online CBT being offered is based on the questionable premise that the children harbor untrue cognitions about their illness.

Third, the standard treatments received by the usual care group were described like this: “individual/group based rehabilitation programs, psychological support including CBT face-to-face, graded exercise therapy by a physiotherapist, etc.” In other words, pretty much the kinds of “evidence-based” strategies these Dutch experts and their U.K. colleagues had promoted for years as being effective for chronic fatigue syndrome. In the end, two-thirds of those in usual care received in-person CBT, and half received graded exercise therapy. (Many participants in this arm received more than one form of usual care.)

And yet less than one in ten of the usual care participants were found to have “recovered” at six months, according to the 2012 study. So what does that say about the effectiveness of these kinds of rehabilitative approaches in the first place? In light of the superlative findings for online CBT, why haven’t all chronic fatigue syndrome patients in the Netherlands now been removed from in-person treatments and offered this more convenient option? (Dr. Crawley’s FITNET-NHS proposal tried to explain away this embarrassing finding of the Dutch study by suggesting that those providing usual care were not trained to work with this kind of population.)

Finally, the Dutch study did not report any objective measures of physical performance. Although the study included assessments using an actometer–an ankle bracelet that monitors distance moved–the Lancet paper did not mention those results. In previous studies of cognitive and behavioral treatments for ME/CFS, reported improvements on subjective measures for fatigue or physical function were not accompanied by increases in physical movement, as measured by actometer. And in PACE, of course, the investigators dismissed their own objective measures as irrelevant or non-objective–but only after these outcomes failed to provide the desired results.

In response to correspondence calling for publication of the actometer data, the Dutch investigators refused, noting that “the goal of our treatment was reduction of fatigue and increase in school attendance, not increase in physical activity per se.” This is an inadequate explanation for the decision to withhold data that would shed light on whether participants actually improved in their physical performance as well as in their subjective impressions of their condition. If the actometer data demonstrated remarkable increases in activity levels in the online CBT group, is there any doubt they would have reported it?

In short, the Dutch FITNET study leaves a lot of questions unanswered. So does its U.K. version, the proposed FITNET-NHS. And Dr. Crawley’s recent media blitz–which included a “can’t-we-all-get-along” essay in The New Scientist–did little to quell any of the reasonable qualms observers might have about this latest effort to bolster the sagging fortunes of the CBT/GET/PACE paradigm.

“Patients are desperate for this trial, yet some people are still trying to stop us,” wrote Dr. Crawley in The New Scientist. “The fighting needs to end.”

However, those mysterious and sinister-sounding “some people” cited by Dr. Crawley have very thoughtful and legitimate reasons for questioning the quality of her research. The fighting, as she calls it, is likely to end when Dr. Crawley and her colleagues stop conflating chronic fatigue and chronic fatigue syndrome through the use of loose diagnostic criteria. And when they acknowledge what scientists in the U.S. and around the world now understand: The claim that cognitive and behavioral approaches are effective treatments that lead to “recovery” is based on deeply flawed research.

A Short Postscript:

Several Dutch colleagues have joined Dr. Crawley as part of the FITNET-NHS study. Two of them, Dr. Gijs Bleijenberg from the Radboud University Medical Centre in Nijmegen, and Dr. Hans Knoop from the University of Amsterdam, are among the leaders of the CBT/GET movement in the Netherlands and have collaborated with their U.K. counterparts. Not surprisingly, their work is similarly dodgy.

In a post last year, I dissected a 2011 commentary in The Lancet on the PACE trial, co-authored by Dr. Bleijenberg and Dr. Knoop, in which they argued that 30 percent of the participants in the CBT and GET groups had met “a strict criterion for recovery.” This statement was absurd, since these “strict” thresholds for “recovery” were in fact so lax that participants could get worse during the study and still meet them. Although the problematic nature of the thresholds has been pointed out to Dr. Bleijenberg and Dr. Knoop, they have stood by their nonsensical claim.

Earlier this year, the Dutch parliament asked the Health Council–an independent scientific advisory body–to review the state of evidence related to the illness, including the evidence on treatments like CBT and GET. The Health Council appointed a committee to conduct the review. Among the committee members are Dr. Knoop and colleagues who share his perspective. It remains unclear whether the committee is taking sufficient account of the methodological flaws underpinning the evidence for the CBT/GET paradigm and of the ongoing condemnations of the PACE trial from well-respected scientists. I plan to blog about this situation soon.