By David Tuller, DrPH
Last November 12, I published a post called “How Bristol Investigators Avoided Ethical Review.” That post addressed the series of eleven studies conducted by Bristol investigators and exempted from ethical review on questionable grounds. The lead investigator of these studies was Professor Esther Crawley, Bristol’s methodologically challenged pediatrician, whose work was the subject of a report released last week by the National Health Services’s Health Research Authority.
That investigation confirmed my argument that these studies all qualified as “research” and not “service evaluation.” Professor Crawley has now been directed to make corrections to the ethics statements in all eleven studies. While this remedy falls significantly short of what I believe is appropriate, it would nonetheless be a humbling and perhaps humiliating task for any serious investigator to have to correct so many studies at one go. And of course this mandate comes just a few months after Archives of Disease in Childhood appended a 3000-word correction to the dung-heap known as the Lightning Process trial–also from Professor Crawley and her colleagues.
I will address my concerns about the new report on Professor Crawley’s work at another time. For now, I thought it might be useful to reprise last year’s post (published in full below). The post offers an in-depth explanation of the issues involved, including the distinction between “research” and “service evaluation.”
**********
November 12, 2018:
I have written many posts about BMJ Open’s 2011 school absence study,
which reported that school absence records could be useful in identifying children with chronic fatigue syndrome. However, for reasons not yet adequately explained, the investigators exempted the study from ethical review on the grounds that it qualified as “service evaluation.” To support the claim, they cited a 2007 letter from the regional research ethics committee (REC), which was unrelated to the study in question.
Studies involving data from human subjects are often categorized as either “service evaluation” or “research.” Those in the former group are generally supposed to evaluate the provision of health care or other services through analyses of anonymous data; they are exempt from ethical review. If a study is testing a hypothesis, produces generalizable conclusions and/or involves data collected from participants known to the investigators, it falls under the category of “research,” not “service evaluation,” and requires ethical review.
The school absence study was conducted by investigators from the University of Bristol. It featured a formal hypothesis, generalizable conclusions, and in-person collection of data from participants. It was clearly “research,” as understood by the Health Research Authority, the arm of UK the National Health Service that serves as the arbiter of research ethics. The study should never have been defined as service evaluation and published without ethical review. Yet BMJ Open has repeatedly defended the decision to publish it.
As it turns out, the 2007 REC letter has been cited to exempt at least ten other studies from ethical review as service evaluation. (In some cases, the claimed exemption was only for part of the study.) Perhaps exemption from ethical review could be justified in some instances. But given the Bristol investigators’ inappropriate decision to cite the 2007 REC letter in exempting the school absence study from ethical review, further scrutiny of these studies would seem to be warranted. (A smart source introduced me to the issues involving the 2007 REC letter.)
*********
Here’s the background:
In 2007, the North Somerset and South Bristol REC approved a request to add questionnaires to those already being used during pediatric care at the Bath CFS/ME clinical service. At the time, the clinical service was conducting assessments with several questionnaires at entry and at twelve months. The REC application sought approval for the Bath CFS/ME clinical service to add further assessments at six weeks and six months, arguing that this would be useful for measuring and improving the delivery of care.
In the documents, the title of the research project associated with the application was: “What happens to children with CFS/ME? The study of a longitudinal cohort of children who access a paediatric CFS/ME service.” After reviewing the application, the North Somerset and South Bristol REC sent a letter dated May 1, 2007. Here’s the operative phrasing from this letter: “Members [of the REC] considered this project to be service evaluation. Therefore it does not require ethical review by a NHS Research Ethics Committee or approval from the NHS R&D office.”
The 2007 REC letter referred to “this project”, i.e., the expanded schedule of pediatric assessments at the Bath CFS/ME clinical service. It did not present itself as a precedent for avoiding ethical review for future, as-yet-undescribed projects. Yet that is how members of the Bristol team of investigators seem to have used it.
Besides the school absence study, here are the other papers (in no particular order) that have cited this 2007 REC letter to claim exemption from ethical review:
https://www.ncbi.nlm.nih.gov/pubmed/19452195
http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-217
http://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-11-308
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3665909/
https://www.ncbi.nlm.nih.gov/pubmed/23619200
http://bmjopen.bmj.com/content/5/10/e008830.full
https://www.ncbi.nlm.nih.gov/pubmed/27655658
https://www.ncbi.nlm.nih.gov/pubmed/19001477
https://www.ncbi.nlm.nih.gov/pubmed/26800634
https://hqlo.biomedcentral.com/articles/10.1186/s12955-018-1028-2
Some of these studies involved data from adults, so it is hard to understand why they would be considered exempt from ethical review based on an REC letter related to routine pediatric questionnaires. Five of the exempted studies were published in BMJ journals, three in Archives of Disease in Childhood and two in BMJ Open, including the school absence study. This doesn’t surprise me. For almost a year, I have been criticizing both journals for lax oversight.
In the case of Archives of Disease in Childhood, our ongoing discussion involves an unrelated study from members of the same team of investigators, a trial of the Lightning Process as a pediatric treatment for children with chronic fatigue syndrome. This trial violated BMJ’s strict policy on prospective registration but is still in the literature. In the case of BMJ Open, the issue has been the school absence study and its lack of ethical review. That study helped trigger my interest in how the same 2007 REC letter was used to exempt other research from ethical review.
**********
Besides the BMJ Open school absence paper, BMJ published four more studies that specifically cited the 2007 REC letter. None seem to meet the criteria for service evaluation of pediatric care.
The second BMJ Open paper on the list involved data from adults as well as children and included this explanation of the finding of the 2007 REC letter: “The North Somerset and South Bristol Research Ethics Committee decided that collection and analysis of CFS/ME patient data constituted service evaluation and did not require ethical review by a NHS Research Ethics Committee or approval by NHS Research and Development offices.”
This expansive reading of the 2007 REC letter would seem to go beyond the meaning and intent of the letter’s finding that “this project”, the collection of specific pediatric data for a longitudinal cohort study–qualified as “service evaluation.” Ethics declarations from other papers in the group use similarly broad terms in describing the 2007 REC letter to exempt themselves from ethical review on the grounds of service evaluation.
The three studies published in Archives of Disease in Childhood and exempted from ethical review on service evaluation grounds do not seem to have focused on the evaluation of pediatric services. Instead, they presented generalizable conclusions about children with the illness.
One of the studies concluded this, for example: “Depression is commonly comorbid with CFS/ME, much more common than in the general population, and is associated with markers of disease severity. It is important to screen for, identify and treat depression in this population.” Studies with generalizable conclusions are normally categorized as “research” and require ethical review.
It is unclear why a narrowly written 2007 REC letter about a particular pediatric research project at one clinical service seems to have been treated as something of an invitation to bypass ethical review in a range of pediatric and adult studies. Bristol University should assess whether each study in this group was or was not appropriately exempted from ethical review based on the finding from the 2007 REC letter. The university should also determine whether members of its faculty commonly use this method to exempt their studies from ethical review.
For its part, BMJ has an independent responsibility to explain why it published multiple studies that exempted themselves from ethical review on questionable grounds, all citing the same 2007 REC letter. Editors need to ask themselves some tough questions. Here are a few:
*What lapses in procedures for assessing and reviewing manuscripts might have allowed these studies to slip through? What red flags were missed?
*What is current policy about fact-checking ethics declarations, especially when they are used to exempt studies from ethical review?
*In the case of the 2011 school absence study, the abstract itself provides sufficient evidence to determine that this was “research” requiring ethical review and not “service evaluation.” Why doesn’t BMJ Open apologize for making an error in judgement and address the issue honestly?
*What steps can BMJ take to prevent such potential embarrassments in the future and to bolster public confidence in the integrity of its editorial decision-making?
Comments are closed.