Update: July 30, 2018
This letter inaccurately explained the issues with the PLoS One study. Specifically, when I wrote it I misunderstood which analyses had and hadn’t been done. The letter was e-mailed to PloS One long ago, so I can’t correct what has already been sent. But the following three paragraphs should have replaced the middle section of the letter:
“The PACE statistical analysis plan included three separate assumptions for how to measure the costs of what they called informal care€“the care provided by family and friends, in assessing cost-effectiveness from the societal perspective. The investigators promised to analyze the data based on valuing this informal care at: 1) the cost of a home-care worker; 2) the minimum wage; and 3) zero cost. The latter, of course, is what happens in the real world, families care for loved ones without getting paid anything by anyone.
In PLoS One, the main analysis for assessing informal care presented only the results under a fourth assumption not mentioned in the statistical analysis plan, valuing this care at the mean national wage. The paper did not explain the reasons for this switch. Under this new assumption, the authors reported, CBT and GET proved more cost-effective than the two other PACE treatment arms. The paper did not include the results based on any of the three ways of measuring informal care promised in the statistical analysis plan. But the authors noted that sensitivity analyses using alternative approaches did not make a substantial difference to the results and that the findings were robust under other assumptions for informal care.
Sensitivity analyses are statistical tests used to determine whether, and to what extent, different assumptions lead to changes in results. The alternative approaches mentioned in the study as being included in the sensitivity analyses were the first two approaches cited in the statistical analysis plan, valuing informal care at the cost of a home-care worker and at minimum wage. The paper did not explain why it had dropped any mention of the third promised method of valuing informal care, the zero-cost assumption.”
**********
PLoS One
1160 Battery Street
Koshland Building East, Suite 100
San Francisco, CA 94111
Dear PLoS One Editors:
In 2012, PLoS One published Adaptive Pacing, Cognitive Behaviour Therapy, Graded Exercise, and Specialist Medical Care for Chronic Fatigue Syndrome: A Cost-Effectiveness Analysis. This was one in a series of papers highlighting results from the PACE study, the largest trial of treatments for the illness, also known as ME/CFS. Psychologist James Coyne has been seeking data from the study based on PLoS’ open-access policies, an effort we support.
However, as David Tuller from the University of California, Berkeley, documented in an investigation of PACE published last October on Virology Blog, the trial suffered from many indefensible flaws, as patients and advocates have argued for years. Among Dr. Tuller’s findings: the main claim of the PLoS One paper–that cognitive behavior therapy and graded exercise therapy are cost-effective treatments–is wrong, since it is based on an erroneous characterization of the study’s sensitivity analyses. The PACE authors have repeatedly cited this inaccurate claim of cost-effectiveness to justify their continued promotion of these interventions.
Yet the claim is not supported by the evidence, and it is not necessary to obtain the study data to draw this conclusion. The claim is based solely on the decision to value the free care provided by family and friends as if it were compensated at the level of a well-paid health care worker. Here is what Dr. Tuller wrote last October about the PLoS One paper and its findings:
The PLoS One paper argued that the graded exercise and cognitive behavior therapies were the most cost-effective treatments from a societal perspective. In reaching this conclusion, the investigators valued so-called informal care, unpaid care provided by family and friends€“at the replacement cost of a homecare worker. The PACE statistical analysis plan (approved in 2010 but not published until 2013) had included two additional, lower-cost assumptions. The first valued informal care at minimum wage, the second at zero compensation.
The PLoS One paper itself did not provide these additional findings, noting only that sensitivity analyses revealed that the results were robust for alternative assumptions.
Commenters on the PLoS One website, including [patient] Tom Kindlon, challenged the claim that the findings would be robust under the alternative assumptions for informal care. In fact, they pointed out, the lower-cost conditions would reduce or fully eliminate the reported societal cost-benefit advantages of the cognitive behavior and graded exercise therapies.
In a posted response, the paper’s lead author, Paul McCrone, conceded that the commenters were right about the impact that the lower-cost, alternative assumptions would have on the findings. However, McCrone did not explain or even mention the apparently erroneous sensitivity analyses he had cited in the paper, which had found the societal cost-benefit advantages for graded exercise therapy and cognitive behavior therapy to be robust under all assumptions. Instead, he argued that the two lower-cost approaches were unfair to caregivers because families deserved more economic consideration for their labor.
In our opinion, the time spent by families caring for people with CFS/ME has a real value and so to give it a zero cost is controversial, McCrone wrote. Likewise, to assume it only has the value of the minimum wage is also very restrictive.
In a subsequent comment, Kindlon chided McCrone, pointing out that he had still not explained the paper’s claim that the sensitivity analyses showed the findings were robust for all assumptions. Kindlon also noted that the alternative, lower-cost assumptions were included in PACE’s own statistical plan.
Remember it was the investigators themselves that chose the alternative assumptions, wrote Kindlon. If it’s ‘controversial’ now to value informal care at zero value, it was similarly ‘controversial’ when they decided before the data was looked at, to analyse the data in this way. There is not much point in publishing a statistical plan if inconvenient results are not reported on and/or findings for them misrepresented.
Given that Dr. McCrone, the lead author, directly contradicted in his comments what the paper itself claimed about sensitivity analyses having confirmed the robustness of the findings under other assumptions, it is clearly not necessary to scrutinize the study data to confirm that this central finding cannot be supported. Dr. McCrone has not responded to e-mail requests from Dr. Tuller to explain the discrepancy. And PLoS One, although alerted to this problem last fall by Dr. Tuller, has apparently not yet taken steps to rectify the misinformation about the sensitivity analyses contained in the paper.
PLoS One has an obligation to question Dr. McCrone about the contradiction between the text of the paper and his subsequent comments, so he can either provide a reasonable explanation, produce the actual sensitivity analyses demonstrating robustness under all three assumptions outlined in the statistical analysis plan, or correct the paper’s core finding that CBT and GET are “cost-effective” no matter how informal care is valued. Should he fail to do so, PLoS One has an obligation itself to correct the paper, independent of the disposition of the issue of access to trial data.
We appreciate your quick response to these concerns.
Sincerely,
Ronald W. Davis, PhD
Professor of Biochemistry and Genetics
Stanford University
Rebecca Goldin, Ph.D.
Professor of Mathematical Sciences
George Mason University
Bruce Levin, PhD
Professor of Biostatistics
Columbia University
Vincent R. Racaniello, PhD
Professor of Microbiology and Immunology
Columbia University
Arthur L. Reingold, MD
Professor of Epidemiology
University of California, Berkeley
Comments are closed.