By David Tuller, DrPH
This is a complicated post. Here are the key points. The rest is details:
*Professor Esther Crawley and co-authors claimed a 2011 study in BMJ Open was exempt from ethical review because it involved the routine collection of data for service evaluation. Yet the 2011 study was not an evaluation of routine clinical service provision–it was designed to road-test a new methodology to identify undiagnosed CFS/ME patients among students with records of chronic absence.
*To support the claim that the study was exempt from ethical review, Professor Crawley and co-authors cited a 2007 research ethics committee opinion that had nothing to do with the data-collection activities described in the 2011 paper.
*For the 2011 study, school letters were sent to families of 146 student, inviting them to meet with Professor Crawley. In the end, only 28 were identified as having CFS/ME–meaning more than 100 families of students without CFS/ME received potentially disconcerting letters inviting them to a medical meeting about a sensitive issue. This type of pilot program is beyond the scope of what many would consider to be service evaluation.
*A pre-publication reviewer, noting the data collection activities described in the paper, raised serious questions about the lack of ethical review. In her response, Professor Crawley did not provide satisfactory answers to the concerns raised by the reviewer, but BMJ Open published the paper anyway, without ethical review.
*BMJ Open‘s recent response to the concerns has been confused, contradictory and inadequate. In separate e-mails, the editor and editor-in-chief have provided two distinct and incompatible justifications for the decision to publish without ethical review. Neither explanation is convincing.
**********
For a 2011 study, Professor Esther Crawley of Bristol University hypothesized that many children remained undiagnosed for chronic fatigue syndrome/myalgic encephalomyelitis. So she decided to investigate whether school absence records could yield further cases.
Working with three schools, she designed a pilot program that targeted students with a history of being absent for unexplained reasons 20 percent or more of the time. Students and their families were invited to meet and discuss this pattern of school absence with Professor Crawley, a pediatrician, along with a school staff member. Some of these students were subsequently evaluated and treated at Professor Crawley’s Bath clinical service for young people with CFS/ME. (I am using CFS/ME here because that is the term Professor Crawley uses.)
The study, published in the journal BMJ Open, concluded that a program involving this kind of school outreach based on absence records could identify previously undiagnosed children, who could then benefit from treatment. But there was a troubling twist: Professor Crawley and her co-authors did not seek ethical review from a U.K. National Health Service Research Ethics Committee, as would normally be expected for studies involving human subjects. Instead, the paper included the creative claim that the study was exempt from such ethical review because it qualified as service evaluation.
[Key information in this post comes from documentation provided by an independent researcher, who obtained it via a freedom-of-information request to the NHS Health Research Authority and then corresponded with BMJ Open about the issue. To be fully transparent, Professor Crawley is not a fan of my work and has publicly accused me of writing libelous blogs. However, she has repeatedly failed to respond to requests that she present documentation or evidence that anything I have written about her work is false or inaccurate.]
According to HRA guidelines, service evaluation studies are designed and conducted solely to define or judge current care and “involve minimal additional risk, burden or intrusion for participants.” For these studies, investigators are not required to seek the kind of ethical review from an REC that is mandated for what is deemed research, that is, studies that are potentially more risky, burdensome or intrusive for participants and raise more possible ethical concerns.
A pre-publication review of the 2011 paper highlighted the lack of ethical review as a major concern. [BMJ Open has an open review process, so reviews and author responses are posted with the published article.] In his comments, the reviewer questioned how the actions taken to identify new patients, as described in the paper, could be considered service evaluation rather than research. In her response to this pre-publication review, Professor Crawley did not provide direct and satisfactory answers to the reviewer’s concerns, yet BMJ Open editors apparently took no further steps to address the issue.
The independent researcher who obtained the documentation wrote to BMJ Open about the issue earlier this year. After reviewing the matter, the journal’s editor acknowledged the obvious, that Professor Crawley’s 2011 study is not strictly a service evaluation–but maintained that it was nonetheless exempt from ethical review for other reasons. This retroactive claim for exemption, however, was itself based on false information.
(I know, I know–it’s confusing. Sorry!)
The central question here is whether the 2011 BMJ Open study should have been defined as research, which would have required REC ethical review, or service evaluation, which would not. BMJ Open itself appeared to have expressed its view by publishing the study under a prominent heading slugged Research. The study included a hypothesis to be tested as well as a specific research question–markers of what is typically defined as research and not as service evaluation.
Moreover, the paper clearly presented itself not as investigating current care, per the HRA definition of service evaluation, but as piloting a new strategy or intervention to identify previously undiagnosed patients. The main outcome measure was the number of children newly diagnosed as having CFS/ME. On the face of it, this does not sound like part of service evaluation involving care for patients already being seen.
To support the claim that the study did not need to undergo ethical review from an REC, the paper provided the following explanation: The clinical service in this study was provided as an outreach from the Bath specialist CFS/ME service. The North Somerset & South Bristol Research Ethics Committee decided that the collection and analysis of data from children and young people seen by the CFS/ME specialist service were part of service evaluation and as such did not require ethical review by the NHS Research Ethics Committee or approval from the NHS R&D office (REC reference number 07/Q2006/48). [The separate issue of obtaining approval from NHS R&D is not of concern here.]
An REC reference number cited in a paper would often identify an opinion specifically about the particular study and data-collection method at issue. In this case, the REC number involved a 2007 decision about a data collection procedure very different from the activities involved in the 2011 pilot program to conduct outreach through the monitoring of school absence records. [The independent researcher obtained the documentation about this 2007 decision under her freedom-of-information request.]
The 2007 REC decision involved an application seeking permission to expand the schedule of assessments of children referred to and receiving specialist care at Professor Crawley’s CFS/ME clinical service in Bath. The clinical service was at that time conducting assessments with several questionnaires at entry and at twelve months. The REC application proposed adding further assessments at six weeks and six months, arguing that this would be useful for service evaluation as well as improving the delivery of clinical care.
In response to a question about how patients for this expanded questionnaire regimen would be identified, approached, and recruited, the REC application declared that there will be no change in the way potential participants are identified. In other words, the application was explicitly not describing or seeking permission for identifying new patients or implementing new methods to recruit undiagnosed children. It was simply seeking permission to collect some more data on patients who were being referred to the CFS/ME clinical service through standard channels. The additional burden to patients and their families was considered minimal; the REC application estimated that filling out the two sets of extra questionnaires would take less than twenty minutes each time.
After reviewing the application, the North Somerset & South Bristol REC sent a letter dated May 1, 2007. Here’s the operative phrasing: Members [of the REC] considered this project to be service evaluation. Therefore it does not require ethical review by a NHS Research Ethics Committee or approval from the NHS R&D office.”
The letter referred to this project, -i.e. the activities proposed in the application, specifically the expanded schedule of assessments that would take participants less than forty minutes to complete. The letter did not indicate that the same consideration or determination applied to other, as-yet-unspecified projects with as-yet-unspecified data-collection activities, -such as efforts to monitor school absences and recruit new patients.
Yet Professor Crawley has cited this 2007 REC reference number to support the case that not just the 2011 study but several other studies were exempt from REC review as service evaluation rather than research. Some of the other studies that cite the REC reference number appear to genuinely qualify as service evaluation. But by the BMJ Open editor’s own admission, Professor Crawley’s 2011 study is not strictly service evaluation, a determination that contradicts what the paper states about itself.
In the 2011 paper, the authors appear to make excessive claims about the 2007 service evaluation exemption provided by the North Somerset & South Bristol REC. The 2007 REC letter did not make a blanket assertion that any data collection from any young people seen by the clinical service could be considered service evaluation. It made the narrower finding that expanding the current assessment schedule by adding several questionnaires at two points in time, as specifically outlined in the REC application, could be considered service evaluation.
Let’s compare that limited scope of activity approved as service evaluation to the data collection strategy pursued for Professor Crawley’s 2011 study. After school attendance officers identified children who met the study’s designated absence threshold, the families were sent a letter from the school that invited them to meet with a paediatrician from the Bath specialist CFS/ME team (EMC) and a member of school staff to discuss why their child was missing school. [EMC is Professor Crawley.] Some of these students were then assessed at the Bath clinical service and offered treatment, if indicated.
It should be noted that the study’s designated thresholds for school absences netted many more students than were ultimately diagnosed as having CFS/ME. Letters were sent to the families of 146 students who met the absence criteria. In the end, 28 of them were identified as having CFS/ME. Let’s put that another way: For this study, the families of 118 students who did not have CFS/ME were sent school letters calling them to a medical meeting about a sensitive issue. The families were sent these letters, which could possibly have caused anxiety and alarm, as part of what was purportedly service evaluation of care for young people already diagnosed with CFS/ME.
It is hard to understand how sending out such letters and recruiting new patients in this potentially intrusive manner could be considered part of service evaluation for current care, especially since this was a pilot project. It seems unusual that researchers would bypass ethical review for such an active patient recruitment effort–even more so given that their approach for identifying possible cases of CFS/ME was likely to impact an unknown number of students and families beyond the specific group of interest.
In fact, a reviewer invited by BMJ Open to comment on the draft of the 2011 paper appeared perturbed at the lack of ethical review and approval. In his comments, he expressed surprise that the REC would have considered the outreach aspects of the study to be service evaluation exempt from ethical approval, rather than research requiring it.
Here’s what he wrote: It is understandable that the REC might see use of routine data from the existing clinical service as not being research€¦but it is surprising that they did not see the surveillance component as research. Children who are unknown to services were being contacted using information from their schools and it seems to me that there are significant issues of confidentiality and data protection which, in my experience as a researcher and one time REC member, I am surprised the REC did not think amount to research. Assuming the REC was fully aware of these issues, and still made a decision that the work was not research, then it would be unfair to oppose publication on those grounds, but the authors should make a fuller explanation, and in the interest of openness might want to make their application to the REC and subsequent correspondence available with the publication.
The reviewer’s statement clearly presumed that Professor Crawley and her colleagues had filed an application and corresponded with the appropriate REC about the specific set of activities involved in this school absence study. Given that presumption, he made a reasonable request–that they should publish the REC application and correspondence along with the paper. The reviewer also requested the authors to provide more information about what the families were told and how their consent to participate in the project was obtained.
A pre-publication review like that should raise red flags with editors–at least enough for them to ensure that the authors provide acceptable answers. That apparently did not happen in this case. In response to the reviewer’s comments, Professor Crawley did not cite any correspondence with the REC about the school absence study under discussion, perhaps because no such correspondence existed. Nor did she offer much detail on what information was provided to families and on how consent was obtained. And she did not mention that she was not relying on recent REC correspondence about this specific study, as the reviewer presumed, but on an REC opinion about a much narrower method for additional data collection from four years earlier.
Instead, Professor Crawley explained in her response that the specialist service has been advised that ethical approval for routine collection and analysis of service data is not required. She did not provide a legitimate explanation for why this pilot program qualified as routine collection and analysis of service data when it involved outreach to families whose children were not already enrolled in the clinical service, including many families whose children did not even have CFS/ME. Instead, she pointed out that she is a community pediatrician and that the children were seen in school clinics, although it is not immediately evident why these facts should have exempted the study from ethical review.
In her response, Professor Crawley also referenced top-level admiration for her work. The project has been of great interest to the Department of Education who included it last year as an exemplar in their training for attendance officers in the UK, she wrote. But whether or not education officials were impressed with Professor Crawley and her work was of course irrelevant to the question at hand, which was whether the study should have been considered research or service evaluation. It was not clear why Professor Crawley included this point, except perhaps to suggest that she had well-placed supporters.
According to Professor Crawley’s response, she sought further assurance that the data collection involved was indeed part of service evaluation. Here’s what she wrote: We checked with the co-ordinator for the local REC that recording outcomes on school based clinics run by school nurses is part of service evaluation (and therefore does not require a submission to Ethics) and they have agreed that it is.
This statement is confusing. The clinics described in this study cannot reasonably be defined as nurse-run school-based clinics. According to the paper, they were set up specifically so Professor Crawley could meet with the children and families identified through the pilot program she developed. Professor Crawley is not a school nurse. The paper’s description of the meetings with families indicated the presence of a member of school staff, not a school nurse. Professor Crawley’s own response to the pre-publication reviewer indicated that this staff member was usually the attendance officer.
In fact, the study itself stated that it would be of interest to evaluate whether school nurses, rather than doctors, can undertake the initial assessments in school clinics. In other words, the way Professor Crawley personally collected data for this pilot program had little or nothing to do with the provision of routine care in nurse-run school clinics.
In any event, Professor Crawley provided no documentation of this exchange with the unnamed co-ordinator for the local REC.” Nor did she provide any details of what this unnamed local coordinator was told. Was the coordinator told simply that this was data collection made during routine nurse-run school clinics? Or was the coordinator told that this was a pilot program to identify previously undiagnosed patients using their school absence records?
Furthermore, was the coordinator told that families would be sent letters inviting them to meet with a community pediatrician at the school, in the likely presence of the school attendance officer but not a school nurse? Was the coordinator told that the students might then be recruited as patients into the clinical service run by the community pediatrician? Was the coordinator told that most of the families identified and impacted by this recruitment process turned out not to have children with CFS/ME? Absent further documentation of this apparently critical exchange, Professor Crawley’s second-hand reassurance that the unnamed local coordinator agreed with her interpretation of events is utterly meaningless.
The independent researcher who drew my attention to this case (and provided input for this post) had sought clarification from BMJ Open about the lack of ethical review. In response to her inquiry, she received an e-mail from editor Adrian Aldcroft in June. He wrote this about the study:
While we appreciate the article published in BMJ Open is not strictly a service evaluation, we agree with the statement provided by The University of Bristol that further ethical approval would not have been required for the analysis due to the following exemption:
‘REC review is not required for the following types of research: Research limited to secondary use of information previously collected in the course of normal care (without an intention to use it for research at the time of collection), provided that the patients or service users are not identifiable to the research team in carrying out the research.’
The data in the BMJ Open article meet these criteria.
As such, we do not think any further action is necessary relating to the article.
This response did not resolve the matter. First, to state that the 2011 article was “not strictly a service evaluation was to validate the complaint. It was also a tacit acknowledgement that the paper’s own claim for exemption from ethical review was unjustified. Then, in a sort of sleight-of-exemption, Aldcroft cited a recent statement from Bristol University that studies were exempt if they involved secondary use of information previously collected in the course of normal care. And Aldcroft then stated flatly that the data in the article “meet these criteria.
This last statement is startling because it so obviously untrue. The 2011 paper did not involve secondary use of data previously collected in the course of normal care. The data were collected through implementation of a pilot program designed by Professor Crawley to answer the specific research question posed at the beginning of the article: Are school-based clinics a feasible way to identify children with CFS/ME and offer treatment?
Moreover, the Bristol University exemption cited by Aldcroft stated that the patients or service users must be not identifiable to the research team. In this case, Professor Crawley was the community pediatrician who met with the families; she was also the head of the research team. Under the circumstances, it is impossible to argue that patients in the study were not identifiable to the research team, per the exemption requirements. It is therefore hard to understand Aldcroft’s statement that the data in the 2011 article meet the necessary criteria for exemption from ethical review. It would have been easy for him to determine the correct answer by reviewing the paper itself.
Last Thursday, I wrote to Aldcroft to follow up. I also wrote to both the local REC and the HRA, and to Bristol University. (I did not write directly to Professor Crawley, since I gather she does not want to receive e-mails from me. Instead, I wrote the legal department representative to whom I complained earlier this year about Professor Crawley’s false accusation that I had written libelous blogs.)
I did not hear back from Bristol. A spokesperson for the HRA wrote that he could not answer my specific questions about this case, but he sent general information about RECs and service evaluation. I did receive an e-mail from Trish Groves, editor-in-chief at BMJ Open (she also has other titles), who wrote that Aldcroft had forwarded my e-mail to her.
Dr. Groves wrote that the 2011 study authors had addressed the concerns about ethical review raised by the peer reviewer. After citing the ethical approval statement included in the 2011 paper itself (and quoted above in this post), Dr. Groves wrote: Given the guidance provided by the local REC, we consider that the authors were entitled to reach the conclusions that they did concerning the need for ethics approval.
This reply is disingenuous and troubling–especially given that BMJ Open itself published the study under the heading “Research.” Dr. Groves has now provided a completely different account of the matter than Aldcroft, the journal’s editor, which indicates some incoherence or confusion in the journal’s position. Aldcroft acknowledged that the paper was not strictly a service evaluation and then provided a retroactive justification for why it was exempt from ethical review anyway. According to Dr. Groves’ version, all was done properly the first time around. But she did not then explain why Aldcroft, on behalf of BMJ Open, had presented an alternate point of view.
Here’s my question for Dr. Groves: Is she really comfortable that–as part of a study defined as service evaluation–more than one hundred families whose children did not have CFS/ME were nonetheless sent school letters on a sensitive issue and invited to meet with Professor Crawley? Does Dr. Groves really believe that testing out a new strategy to identify patients unknown to the clinical service qualifies as service evaluation for routine care? I doubt she actually does believe that, but who knows? Smart people can convince themselves to believe a lot of stupid things. In any event, in dismissing these concerns, BMJ Open has demonstrated that something is seriously amiss with its ethical compass.
Dr. Groves did leave the door open a tiny crack. She wrote: BMJ Open is a member of the Committee on Publication Ethics (COPE) and follows its best practice guidance and policies. In light of the matters that have been raised, we will submit this case (anonymised, as always) to the COPE Forum.
Leading scientific journals are members of COPE. Unfortunately, when it comes to rigorous and honest assessment of research from members of the GET/CBT ideological movement, including Professor Crawley, leading scientific journals have exhibited little in the way of publication ethics.” So I am not optimistic that an examination by COPE will produce better results.
Comments are closed.