By David Tuller, DrPH
Professor Esther Crawley, Bristol University’s methodologically and ethically challenged pediatrician and star researcher, has weighed in with an interim analysis of FITNET-NHS, her trial of online cognitive behavior therapy for adolescents with what she calls CFS/ME. Poor Professor Crawley! Despite her ambitious goal of enrolling a whopping 734 participants, early recruitment fell way below expectations. She is now pursuing a scaled-down, shrunken version–MINI-FITNET-NHS, perhaps?
The interim analysis–Recruiting Adolescents With Chronic Fatigue Syndrome/Myalgic Encephalomyelitis to Internet-Delivered Therapy: Internal Pilot Within a Randomized Controlled Trial, was published last week in the Journal of Medical Internet Research. As the report makes apparent, Professor Crawley’s reach far exceeded her grasp. She and her team failed to deliver on the targets outlined in their trial protocol, which formed the basis for their approvals and funding.
The investigators forecast that FITNET-NHS would receive 286 out-of-area referrals in the first 12 months of recruitment, and that 156 patients, or 55%, would eventually enroll. (I find the relevant sentence to be very confusing, but I think that is what it means. Here it is: “In advance of study launch, recruitment projections estimated 286 out-of-area referrals by the end of the first 12 months, expecting 19.9% (57/286) of these to be ineligible and to recruit 67.8% (194/286) of potentially eligible referrals, which would be 156 recruits.”)
Instead, for the year starting November 2016, the study received 193 such referrals, out of which only 89, or 46%, enrolled. Of those enrolled, only 76 provided data at six months. This dramatic short-fall creates mince-meat out of the proposed analyses.
The new paper follows the publication last December of an amendment to the FITNET-NHS protocol. The amendment explained the disappointing recruitment situation and provided a downgraded proposal with recalculated numbers. If the investigators could recruit 314 participants–a target they deemed feasible based on the experience to that point–the analysis could still yield statistically significant albeit watered-down findings, according to the protocol amendment.
The initially proposed sample size of 734 participants would have provided, per the amendment, “97% power at 1% significance to detect a 0.35-SD difference” between the groups on physical function, the primary outcome as assessed using the SF-36 questionnaire. Instead, the revised size would provide “90% power at 5% significance to detect a 0.4-SD difference.”
A related question involved the numbers needed to do a sub-group analysis of patients with co-morbidities. The original sample size would have provided “80% power to detect a 0.4-standard deviation (SD) difference at 5% significance” between treatment groups within this sub-group. The revised version would provide “53% power at 5% significance to detect a 0.4-SD difference.”
Government funders committed about a million pounds for FITNET-NHS. Despite the reduced quality and precision of the forthcoming analyses, it does not seem likely that UK taxpayers will receive any sort of refund for the amount invested in this massive misstep.
********
What Went Wrong with FITNET-NHS?
What went wrong?
When I first wrote about FITNET-NHS four years ago, I reviewed the protocol and suggested the study was pretty much useless. It was the latest in a stream of open-label trials relying on subjective outcomes, a research design known to be fraught with unknown amounts of bias. Professor Crawley’s decision not to use actimeters of some sort to obtain objective assessments of improved movement and function is hard to understand from a scientific perspective. Now it appears that she badly misjudged public enthusiasm for what she’s selling.
In the new study, Professor Crawley and her colleagues observe that an initial wave of interest, sparked by news coverage, was “well above projected figures” but receded after six months. This pattern of waxing and waning interest did not conform to their expectation of an initial lag in referrals that would build up over time. Even so, it is not clear how the unanticipated referral pattern could account for the poor overall recruitment rate across the full 12-month period. The investigators further appear to blame bureaucratic obstacles involving funding for patients from the €œdevolved€ nations of Wales, Scotland and Northern Ireland.
Here’s what they write:
“Projections included an initial lag phase leading up to 35 out-of-area referrals per month. National media coverage at study launch resulted in a surge of referrals (well above projected figures), which waned 6 months later, reducing to under half of the projected out-of-area referrals per month. This had a knock-on effect on recruitment, which suffered a lag of 6 months into the trial.
As described, funding pathway issues prevented patients from Wales from accessing the trial. Similar funding pathway issues existed with other devolved nations (Scotland and Northern Ireland). Due to the distance from the center, these nations were not likely to refer patients for routine (face-to-face) clinical treatment at the service, and because of funding pathway issues, plans to promote the remote treatment opportunity across these regions to gain referrals were not able to proceed, which reduced the pool for potential referrals.
These rationalizations notwithstanding, the underwhelming numbers do not provide evidence of much public faith in Professor Crawley’s repeated pronouncements about the effectiveness of the FITNET intervention. And yet she and her co-investigators conclude that €œrecruiting young people (and their parents or caregivers) into an RCT of web-based treatment via remote methods is feasible and acceptable.€
*****
A High-Profile Launch for FITNET-NHS
FITNET-NHS and Professor Crawley’s recruitment drive got the kind of high-profile launching pad that few studies receive, a press conference at the UK’s Science Media Centre. At the event, Stephen Holgate, a prominent professor of immunopharmacology at the University of Southampton and the chair (then and now) of the CFS/ME Research Collaborative, praised FITNET-NHS as €œhigh-quality research.€ Since close scrutiny of the study details should have raised concerns about its deficiencies, Professor Holgate’s remarks suggested he had been misled by the hype and hoopla surrounding the research project.
In presenting the trial to the media, Professor Crawley stated that two-thirds of kids who received the FITNET intervention recovered. These claims were based on inflated reports of success from the Dutch FITNET trial that served as a model; The Lancet published these first results in 2012. Professor Crawley did not mention that the Dutch investigators used a post-hoc definition of €œrecovery,€ so it was not difficult for them to select loose criteria that would ensure a high percentage of purportedly €œrecovered€ participants. She did not mention that the purported benefits of the intervention were not apparent at long-term follow-up.
Nor did Professor Crawley mention that those who achieved €œrecovery€ in the Dutch study did not demonstrate better physical function at the end of the trial, according to objective assessment of their movements as measured by actimeter. These data were not included in the report of the trial results but in another paper years later. The failure to report the null results from an objective measure alongside positive results for subjective assessments is a violation of key scientific norms and arguably constitutes research misconduct. The Lancet should never have published the Dutch study without insisting on the inclusion of the missing objective data.
Two key Dutch investigators, one of whom was involved in the initial FITNET study, are also collaborators with Professor Crawley on FITNET-NHS. Since the actimeter measure did not work out favorably for the Dutch team, it should not be surprising that Professor Crawley dispensed with it altogether for her version. Why risk bad news by including an objective measure that might contradict attractive subjective outcomes?
When Professor Crawley introduced FITNET-NHS, the news media parroted her sunny assertions about the online CBT intervention. In an online report, the BBC declared that the intervention €œsuccessfully treats two-thirds of children with chronic fatigue syndrome.€ In a subsequent correction, the news organization noted that €œthis story has been amended to clarify that the therapy is of potential rather than definitive help to children.€ The correction did not mention that the BBC reporter who covered the news promoted by the Science Media Centre was simultaneously serving as a Science Media Centre adviser.
The failure of FITNET-NHS to meet recruitment goals, for whatever reason, suggests that patients and their families might be skeptical about Professor Crawley’s claims and her approach to research. It is possible some were troubled when she declared in an interview around the same time that the PACE trial was a €œgreat, great€ study. Since the PACE trial’s flagrant flaws had become better known among patients by that point, Professor Crawley’s transparently ridiculous praise could have caused some to question her integrity and credibility.
If that’s what happened, Professor Crawley’s more recent research misadventures have likely confirmed their concerns. These would include last year’s 3,000-word correction to a paper that BMJ published even though Professor Crawley and her colleagues violated inviolable research principles. Professor Crawley has also been required to correct the ethics statements in eleven other papers, half of them in BMJ titles.
In any event, after Professor Crawley and her FITNET-NHS team redid their figures and substantially downgraded the robustness of their projected results, they received approval from their various oversight committees of pooh-bahs to trim back the trial. Recruitment has also been extended till October of this year to ensure they reach their greatly reduced goal. Here is how they tried, at the end of their protocol amendment, to put a positive spin on their humiliating recruitment failure:
“With our revised recruitment target of 314 participants, the FITNET-NHS is still set to be the largest paediatric CFS/ME treatment trial in the UK and globally, the results of which will should the future of paediatric CFS/ME treatment delivery.” [sic]
And in response to that: 1) A trial can be “the largest” and still be crap. 2) Professor Crawley and her team did not read this important trial document carefully enough to make sure it ended with a properly constructed English sentence. As it is, we are left wondering about the predicted impact of the study results on “the future of paediatric CFS/ME treatment delivery.” Such sloppiness, and the failure of anyone involved to notice or correct this glitch since last December, is not a propitious sign.
Comments are closed.