Trial By Error: More on the Mahana Mess…

If you appreciate this piece and/or my work in general, here’s a link to my current UC Berkeley crowdfunding effort: https://crowdfund.berkeley.edu/project/20159

**********

By David Tuller, DrPH

Today I sent a third letter to Dr Mel Heyman, a prominent gastroenterologist at the University of California, San Francisco. Dr Heyman is one of Mahana Therapeutics’ science advisers. I’ve been trying to understand why the company decided to license a web-based cognitive behavior therapy program for irritable bowel syndrome. After all,  a major study documented that the program did not produce clinically significant benefits in symptom severity over treatment-as-usual.

As I have previously noted, Mahana has implicitly acknowledged my criticisms by removing its laughable description of the symptom severity reductions as “dramatic” and “potentially game-changing.” But marketing strategy isn’t the problem if a program doesn’t perform as promised. At this point, Mahana’s science advisers need to make it clear whether they believe this program has clinically significant benefits  in reducing symptom severity or not.

**********

Dear Dr Heyman–

In February, before coronavirus swamped everything, I sent you two letters concerning Mahana Therapeutics’ expansive claims about its new web-based cognitive behavior therapy program for irritable bowel syndrome. I also reached out to other Mahana gastroenterology advisers as well as Rob Paull, the chief executive officer. (I have cc’d Mr Paull on this letter, as well as Vincent Racaniello, the Columbia microbiologist who hosts my posts on Virology Blog.)

Now I’m trying to get back to work.

Let’s recap: Mahana announced in January that it had licensed the web-based CBT program from King’s College London. My concerns involved statements in a press release and on Mahana’s website that clearly exceeded the data from the relevant study. The press release described the web-based program’s impacts on symptom severity as “substantial” and “durable.” The website called them “dramatic” and “potentially game-changing.”

As I have repeatedly pointed out, these descriptions cannot be justified. At 12 months, the reported benefits of the web-based program over treatment-as-usual on the study’s measure of symptom severity were statistically significant but clinically insignificant. At 24 months, these reported benefits were neither statistically nor clinically significant. Given the weak results for this core indicator, it is hard to understand why Mahana decided to license the product.

It now appears that Mahana has changed the website’s description of the program’s purported benefits and no longer characterizes them as “dramatic” and “potentially game-changing.” I am pleased to see this change. However, the site currently states that “patients enrolled in a minimal-contact digital CBT program experienced significant and clinically meaningful reduction in the severity of their IBS.”

This is an empty statement, since it would be true if only two patients in the web-based arm had achieved this “significant and clinically meaningful reduction” in symptom severity. It could also easily be said about the treatment-as-usual arm, since at least two patients in that arm also achieved “significant and clinically meaningful reduction” in symptom severity. The relevant question is whether the program provides clinically meaningful benefits over and above what is achieved through treatment-as-usual–and the main results for the symptom severity scale suggest that it does not.

The two data points Mahana includes on the website about these purported benefits are not the central findings for IBS symptom severity. To cite them in this manner is highly misleading, as I have previously explained on Virology Blog.

According to Mahana’s website, “66% of patients reported significant and clinically meaningful reduction in the severity of their IBS.”

It is true that 66% of those in the web-based CBT arm who responded at 12 months had a reduction in the scores on the IBS Symptom Severity Scale of 50 or more points–the threshold for clinical significance. But it is not true that most of those changes can be attributed to the web-based program, which is what the statement implies. Mahana does not mention that 44% of those in the treatment-as-usual arm who reported at 12 months also had a reduction in score of 50 or more on the same scale. Given those numbers, it seems likely that many or most of the 66% in the web-based arm could have reported those improvements anyway.

Moreover, the site does not make clear that only 70% of the study sample provided data at 12 months. We can’t know what the final results for the remaining 30%–those considered “lost to follow-up”–would have been. That means we have no idea how those who dropped out from the web-based program arm felt about the intervention or whether it helped them. At 12 months, as I have already noted, the mean score for the web-based group in the intention-to-treat analysis was 35.2 points lower than for the treatment-as-usual group, quite a bit less than the 50-point difference that is considered a clinically significant improvement.

Mahana’s website also states: “On average, reduction in IBS severity was twice that of patients receiving medical care as usual.”

Again true, and again misleading. When improvements are small, improvements that are twice the size are also small. Just because something doubles does not mean the change is of much or any clinical significance. The more telling statistic is often not the relative difference between groups–the kind cited in the above claim about average reductions of IBS severity–but the absolute difference. In this case, as I’ve already noted, the absolute difference in score between the means of the groups was 35.2 points–well under the 50 points that is considered a clinically significant change.

Unfortunately for Mahana, this web-based CBT program cannot be accurately marketed as having clinically meaningful impacts on symptom severity beyond treatment-as-usual. The coronavirus epidemic certainly heightens the potential appeal of effective web-based therapies, but the data from this study cannot be twisted to mean what they don’t.