Trial By Error: Lead Author of Cochrane’s New Bias Guideline is LP Study Co-Author

By David Tuller, DrPH

Jonathan Sterne, a professor of medical statistics and epidemiology at Bristol University, is the corresponding author of Cochrane€™s revised €œrisk of bias€ tool, which BMJ published online on August 28th. Whatever the merits or defects of this revision, Professor Sterne€™s involvement as the first of more than two dozen authors has to be considered unfortunate.

(Added: On the defect side, this revision makes it easier for unblinded studies relying on self-reported outcomes to be assessed as having “low risk” of bias. In fact, studies with this design are at high risk of bias. Yet they are favored by members of the CBT/GET ideological brigades. More on that in a later post.)

Professor Sterne has co-authored 11 papers with his Bristol colleague, Professor Esther Crawley, according to a search on PubMed. I have documented that at least two of these papers breached core principles of scientific inquiry–the Lightning Process trial published two years ago by Archives of Disease in Childhood, and a 2011 study in BMJ Open that exempted itself from ethical review on specious grounds. Neither article would likely have been published had the journals involved conducted sufficiently rigorous peer review to identify the disqualifying flaws.

[I should disclose here that Professor Crawley has publicly accused me of libel, defamation, and engaging in behavior warranting police intervention. The Bristol vice-chancellor’s office has made at least three complaints about me to the Berkeley chancellor but has provided no apparent evidence of wrong-doing or inaccuracy. I note that I remain an academic appointee in good standing at Berkeley’s Center for Global Public Health, and I continue my investigation with the support of my department and academic colleagues.]

In the conduct and reporting of the Lightning Process trial, Professor Crawley, Professor Sterne and their co-authors recruited more than half of their sample before registration, swapped primary and secondary outcome measures after gathering data from these early participants, and then failed to mention these salient details in the published paper, as I documented shortly after it was published. Last month, Archives of Disease in Childhood, a BMJ journal, posted a correction/clarification of more than 2800 words, along with a lengthy editor€™s note seeking to justify the decision to re-publish–rather than retract–the primary results.

The journal’s action downplayed the significance of the Bristol team’s misbehavior and ignored the extent to which Professor Crawley, Professor Sterne and their co-authors had biased their reported findings with their outcome-swapping. This deficient editorial response dismayed many who care about research integrity and the well-being of children. That’s why dozens of scientists, academics and other experts, including several of my Berkeley colleagues, signed Virology Blog’s recent open letter to Dr Fiona Godlee, BMJ’s editorial director. The letter called the decision to republish the original Lightning Process trial findings “scientifically and ethically indefensible.”

As for BMJ Open’s school absence study, the three authors–including Professor Crawley and Professor Sterne–exempted it from ethical review on the false grounds that it qualified as “service evaluation.” (Virology Blog documented this two years ago, thanks to a reader’s tip). Service evaluation, which is exempt from ethical review, mostly relies on anonymous databases to evaluate the provision of specific services to clients or patients. Under widely accepted criteria, if a study features a hypothesis, generalizable conclusions, and/or non-anonymous data collection through interviews or other means, it qualifies as “research” and cannot be exempted from ethical review as “service evaluation.”

Experienced researchers, including Professor Sterne and Professor Crawley, should be well aware of this distinction between “research” that requires ethical review and “service evaluation” exempt from it. Since the school absence study included a hypothesis, generalizable conclusions, and in-person interviews conducted by Professor Crawley, it did not meet any standard criteria for service evaluation. It is therefore hard to understand why the authors, including Professor Crawley and Professor Sterne, would choose to exempt this study from ethical review and publish it as service evaluation. (It is also hard to understand why BMJ Open editors would stonewall, make untrue claims, cast doubts on my reporting, and deny the obvious when confronted with the incontrovertible facts.)

Bristol University is currently conducting an investigation into the school absence study and a number of others that were similarly exempted from ethical review as service evaluation. This investigation was initiated based on concerns I raised with the Health Research Authority, an arm of the National Health Service. I hope to hear something about the results of this investigation in the near future.

It is indisputable that studies involving hypotheses, generalizable conclusions, and in-person interviews with more than 100 children and/or their family members cannot reasonably be defined as service evaluation and exempted from ethical review. But as I have discovered in the last few years, investigations conducted by people enmeshed in the same systems that generated the problematic research in the first place often serve to white-wash, obfuscate or justify bad behavior rather than expose it for what it is.

In any event, until Professor Sterne acknowledges and explains his own role in the conduct and reporting of these two unpalatable studies, it is hard to take seriously any pronouncements about how to assess €œrisk of bias.€ Here’s what I’d like to ask, as a start:

Why did Professor Sterne sign off on the Lightning Process trial paper, in which critical information about retrospective registration and outcome-swapping were withheld from the public version of events? Is Professor Sterne aware that prospective registration and pre-designated outcomes are considered essential in reducing the “risk of bias”? Regarding the school absence paper, does Professor Sterne really believe that a study featuring a hypothesis, generalizable conclusions, and in-person interviews conducted by the lead investigator can reasonably be defined as “service evaluation” and appropriately exempted from ethical review?

These are legitimate questions not only for Professor Sterne but for Cochrane, BMJ, and Professor Sterne’s many co-authors on this “risk of bias” revision. Given that both the Lightning Process and school absence studies are fraught with the kinds of methodological and ethical irregularities that should be obvious to first-year epidemiology students, it is unclear how Professor Sterne can currently serve as a credible authority on anything.

Comments are closed.

Scroll to Top