Trial By Error: The HRA’s Letter about the Investigation of Bristol Research

By David Tuller, DrPH

Last week, the National Health Service’s Health Research Authority published a long-awaited (by me) investigation of 11 studies from Bristol University. All 11 studies were spearheaded by Professor Esther Crawley, the University of Bristol’s high-profile pediatrician. I had flagged these studies as problematic because in every case the Bristol team exempted them from ethical review on the questionable grounds that they constituted “service evaluation” rather than “research.”

This “independent” investigation confirmed the obvious–the ethics statements were inadequate to describe the activities involved, as I had pointed out. To be clear, a shrewd sleuth initially sussed out these irregularities and alerted me to the issues; again, my work has piggy-backed on other people’s discoveries.

I have many thoughts about this investigation and the inadequacy of the recommended remedies. For now, however, I am posting the letter I received from the HRA at the same time the agency sent me the report itself. As the letter requests, I am posting the text in full rather than selected excerpts. (The formatting of the text in WordPress differs somewhat from the formatting of the original letter. Because of my lack of technological prowess, I’m not sure how to reproduce the image of the letter in the blog. The phrases below in bold replicate the use of boldface in the letter itself.)

**********

22 October 2019

Dear Dr Tuller,

Ref: 18.TP.01
Re: Independent Panel Review of 11 publications on chronic fatigue syndrome and myalgic encephalomyelitis
REC reference: 07/Q2006/48
Study title: What happens to children with CFS/ME? The study of a longitudinal cohort of children who access a paediatric CFS/ME service. Version 2.

We wrote to you on 31 January 2019 responding to a number of concerns you raised regarding Research Ethics Committee (REC) approved studies relating to Myalgic Encephalopathy (ME) and Chronic Fatigue Syndrome (CFS). One concern regarding the school absence study (07/Q2006/48), remained outstanding at the time of our previous letter. This is summarised below:

€¢ Concern that the methodology followed in this study with regard to data collection procedures did not match those procedures as set out in the protocol and should not therefore have been classed as a service evaluation by the REC.

€¢ Query why the journal accepted the claim that investigating a pilot intervention to identify new patients could be exempted from ethical review as a service evaluation.

€¢ Query regarding whether journals should accept, at face value, the claims of authors, RECs and university committees or whether independent judgement should be exercised by journals in making a decision.

We were unable to respond to your concerns previously as an independent review of the publications relating to this study was being conducted. The independent review has now been completed.

Please find enclosed the report from the independent panel, jointly commissioned by the Health Research Authority (HRA) and the University of Bristol, to review eleven publications on chronic fatigue syndrome and myalgic encephalomyelitis which all cited the NHS Research Ethics Committee (REC) reference 07/Q2006/48. Please note personal identifiable information has been redacted from the report as required by the Data Protection Act and General Data Protection Regulation.

The panel€™s purpose was to determine as follows:

a) Whether the collection and analysis of the data referred to in each publication was covered by an NHS Research Ethics Committee ethical review (‘ethical review’) and favourable opinion, including whether there was any departure from a reviewed project that was not covered by additional ethical review.

b) Whether the collection and analysis of the data referred to in each publication was part of service evaluation (either under HRA guidance or in line with National Outcomes Database practice), and therefore did not require ethical review;

c) If not, whether ethical review for the collection and analysis of the data (i) was required at the time of publication and (ii) would be required under current Governance Arrangements for Research Ethics Committees (GAfREC) if published today;

d) Whether further action should be considered in relation to any publication based on the outcome of this review.

The membership of the panel was jointly agreed by the HRA and the University of Bristol and was made up of two members of academic staff from the University of Bristol and an independent chair nominated by the HRA who was a clinical academic from another Russell Group university. The panel members had no prior connection to the research or the research team and were appointed based on their specific experience and expertise in relation to ethics of research into health-related matters.

The HRA is satisfied the panel was suitably independent with no material conflicts of interest, had the appropriate expertise to assess the publications and conducted its review in a fair, impartial and objective manner.

In light of the controversy surrounding this area of research, we agreed at the outset that the anonymity of the panel members would be preserved and that the actions arising from this investigation would form the basis for publication. The HRA will therefore put a statement on our website aligned to the content of this letter. We would request that extracts from this letter are not published. If it is published, it should be published in its entirety so that it reflects a full interpretation of the findings.

Teresa Allen and Professor Sir Jonathan Montgomery met with members of the University team responsible for clinical research on Monday 19 August to hear the outcome and recommendations from the panel€™s review. Please be advised the HRA has accepted all of the panels€™ recommendations as set out in the report and summarised below.

Before it looked at the publications, the panel considered all of the available communications between Professor Crawley and the REC which took place prior to the submission of her first application which had the REC reference number 07/Q2006/48, also known as the School Absence Study.

One critical point to note is that the correspondence documents held in the HRA HARP system made available to the panel were an incomplete record for this study. We only currently therefore have access to the research protocol, application and communications sent by the REC to applicants for this study. In 2006, most documents issued to the National Research Ethics Service were hard copy paper format and the policy at that time only required retention of documents for 3 years following a favourable opinion unless there was a reason to hold onto them.

A communication between the REC and Professor Crawley seeking three points of clarification was not therefore available for review and the panel had to make assumptions based on the response letter from the REC to Prof Crawley. We know that this communication took place but neither Professor Crawley nor the HRA has this original communication. The response letter was available within the document pack. It was clear from the response that Professor Crawley had been planning to undertake clinical assessment work which would lead to the establishment of a National Outcomes database for children with CFS/ME and she was seeking advice from the REC on how existing and new data points should be treated.

The REC, (formerly known as North Somerset and Bristol) had reviewed the study application and concluded that the work that she planned to undertake was in fact a Service Evaluation. The wording used in the letter of response by the REC was considered by the panel to be open to mis-interpretation. One interpretation being that all work undertaken using data from the National Outcomes Database was covered by this original REC decision. Further wording in the letter could also be taken to mean that the REC was giving permission for further work to be undertaken using this data without submitting new applications to the REC for a follow up opinion.

To this effect, the panel found that the REC communications were insufficient and the HRA has already made changes to the way that we produce REC minutes and written communications to applicants to ensure that this type of confusion will not arise in future. Our current processes also require all enquiries regarding studies to be submitted in writing to the Chair so that they can be considered together with submissions. We are also currently in the process of updating our archiving policy to ensure documents are held for an appropriate length of time.

The key findings of the review are listed below:

a) The REC documentation was incomplete and was in some respects lacking in clarity and open to interpretation;

b) No favourable ethical opinion was obtained in respect of any of the papers reviewed;

c) In all eleven papers, the REC reference, as cited in the ethics statement, was
either inappropriate or inadequate to cover the activity described;

d) However, no additional ethical review was required and the position could in all the cases considered be rectified by rewording and clarification of the ethics statement;

e) In two papers, additional minor textual changes in the main body of thepublication would improve clarity.

The panel then considered each of the 11 publications in turn against the original complaint that the publications were describing research and not service evaluation. This was assessed against the HRA published toolkit (2013) to determine whether published work is research or not and the Governance Assurance Framework for Research Ethics Committees (GAfREC 2001) which was valid at the time that each piece of work was undertaken together with more recent versions of this framework.

The view of the panel concurs with our view in the HRA that all 11 of the publications contain material which would classify them as research.

However, a critical point to note is that not all research requires the submission of an application to an NHS REC. Research which includes the use of anonymised patient data is excluded. It is for this reason that the panel have concluded points (b) and (d) above.

The panel have however concluded that the REC reference cited was either inappropriate or inadequate to cover the work described in the publications (c) and in the case of 2 papers that clarity could be achieved by changing some of the text.

The panel were not asked to comment on any of the research findings.

Actions to be taken by Professor Crawley:

Professor Crawley will now be asked to submit amended text to the publishers for each of the 11 publications to include some alternative wording replacing the service evaluation statements and points of clarity for two publications. She will also be asked to notify the funding bodies that these changes are being submitted and to notify us when these changes have been made.

We have also sought the assurance from the University and Professor Crawley that the service evaluation reference 07/2006/48 will no longer be cited in any future publications.
Having considered these findings, we are satisfied that the HRA has already implemented changes which will avoid a recurrence of any mis-interpretation of the REC opinion and our archive policy is being updated so that we will include scanned copies or uploaded correspondence from applicants in our archives.

We are satisfied that Professor Crawley took the correct course of action at the outset in terms of engaging with the Ethics Committee to seek clarity about the original work that she planned to undertake and that the ambiguous wording of the REC communication has generated a number of publications with inappropriate references with respect to the work which followed.

We are also satisfied that she either had, or did not need to seek, additional ethical opinions for the published work reviewed by the panel.

It is clear therefore that there is no case to be made for any disciplinary action against Professor Crawley with respect to this investigation. There are however a number of lessons which can be learned by both parties.

We are satisfied the first of your concerns is addressed above, namely:

€¢ Concern that the methodology followed in this study with regard to data collection procedures did not match those procedures as set out in the protocol and should not therefore have been classed as a service evaluation by the REC.

With regard to your remaining concerns, as detailed below, we are unable to advise any further as these are directed to journal and journal editors and as such beyond the HRA€™s role and remit.

€¢ Query why the journal accepted the claim that investigating a pilot intervention to identify new patients could be exempted from ethical review as a service evaluation.

€¢ Query regarding whether journals should accept, at face value, the claims of authors, RECs and university committees or whether independent judgement should be exercised by journals in making a decision.

We trust this addresses all of the concerns you have raised however if you remain dissatisfied you may wish to write to the Health Service Ombudsman.

The Parliamentary and Health Service Ombudsman Millbank Tower
Millbank
London SW1P 4QP
Tel: 0345 015 4033
Website: www.ombudsman.org.uk