Retraction of two high-profile papers raise concerns about research integrity

Chronology of events

On May 1, the NEJM published a study that suggested that two types of blood pressure medications (angiotensin-converting–enzyme inhibitors and angiotensin-receptor blockers) were not associated with worse outcomes in patients with COVID-19, while noting ACE inhibitors in particular had significantly protective effects on mortality.

On May 22, the Lancet published an article by the same authors that found that chloroquine and hydroxychloroquine, explored as potential therapies for COVID-19, did not correspond with improved outcomes and may be associated with higher mortality by up to 30%.

Over the following week, a number of high-profile clinical trials were halted or suspended as a result of the Lancet paper. That included the multi-country Solidarity Trial that was being coordinated by the WHO.

On May 28, more than 180 scientists submitted an open letter to the Lancet editor and study authors outlining questions about the statistical analysis and a lack of transparency in the underlying data, including contributing countries or hospitals.

On June 2, the editors of the NEJM and Lancet issued respective “Expressions of Concern” over methodological and data integrity questions related to the two publications, while the WHO resumed the relevant arm of the Solidarity Trial. Two days later, the co-authors retracted both papers from the NEJM and Lancet, as the journal editors indicated that they would undertake an internal audit of their procedures.

What are the issues?

Data Integrity

Much of the controversy has been around the quality and nature of the large proprietary collection of electronic health records analysed and provided by Surgisphere, a private medical data provider based in the U.S. The primary author of both papers claimed that an independent audit of the data was requested by Medical Technology and Practice Patterns Institute, but were told by the medical data research institute that Surgisphere would not share the underlying data due to “agreements with its clients and the fact that the documents contain confidential information,” It should be noted that Surgisphere Chief Executive was a co-author on the papers. According to a statement on the Surgisphere company website, its data registry was based on electronic health records from customers of its machine learning program and data analytics platform producing data that allows researchers to study “real-world, real-time patient encounters.”

Methodological integrity

The high-profile retractions and pulled pre-print raise broader questions about working with large datasets that researchers are not able to validate, and particularly about how such work could pass peer review at prestigious medical journals.While it is not uncommon for studies to use large datasets without external scrutiny of the raw data, but it is far more uncommon for high-impact journals like the Lancet. While its editor and the NEJM claim the review process was confidential, both journals claim the papers had “external peer review, statistical review, as well as scientific and manuscript editing.” How such a comprehensive dataset could have been gathered from hospitals around the world in the middle of a pandemic is a question that researchers, journals and peer reviewers should have raised.

Adverse Impacts

Although the retrospective analysis was not a randomized control trial, it received a lot of attention given its publication in a high-impact journal and purported scale of almost 100,000 patient records from 671 hospitals around the world between December 2019 and April 2020. Even if the results were correct, however, experts suggest such observational data, with its inherent weaknesses, should not be used to stop trials that could provide definitive and actionable answers. Among the major adverse impacts of the published Lancet study has been suspension of several well-designed clinical trials. One of the major concerns is that the publicity around the Lancet findings have made it harder to conduct trials or recruit patients, as the retraction won’t receive as much attention as the original study. In retrospect, researchers and decision makers may have placed too much reliance on the study, particularly as there had been prior queries about the methods of analysis and some of the results did not fit with what is known about other risk factors. The Annals of Internal Medicine, for example, maintained an ongoing and regularly updated review that summarised data from randomised trials and from observational studies. The journal did not include the Lancet study in their review or conclusion that “at this time, there is insufficient and often conflicting evidence on the benefits and harms of using chloroquine  or hydroxychloroquine to treat COVID-19” that currently make it “impossible to determine the balance of benefits to harms.”

Ensuring research integrity during a pandemic

Scientific publication must above all be rigorous and honest. The retractions raise larger issues of reproducibility and scientific integrity, particularly when using big databases to draw conclusions, an approach that has been gaining rigor in the era of big data. While big databases have opened new lines of inquiry, there are also major issues as experts warn conducting studies properly is far more difficult and require a thorough understanding of underlying data to ensure the methods are rigorous and outcomes reasonable. There have been calls from scientists for independent auditing of data by journals and possibly regulatory bodies like the FDA which has extensive experience of auditing clinical data.

Part of the challenge has been pace. Unlike prior pandemics like SARS and Ebola, the COVID-19 pandemic has led to substantially more scientific publications, most notably pre-prints that are often not peer-reviewed. The rapidly increasing number of publications and urgency to quickly understand the virus present significant challenges in ensuring that research is conducted according to methods and standards of research integrity. This includes, for example, underlying models used to plan pandemic response and predict future outbreak trajectories that are often “poorly reported, at high risk of bias, and reported performance that are optimistic.” Serological tests that have received accelerated approval raise similar concerns, as claims by manufacturers around specificity and sensitivity often lack published data, presenting challenges as countries use testing to determine how to ease lockdowns.

Related papers

• Top journals raise concerns about data in two studies related to COVID-19 (STAT)
• High-profile coronavirus retractions raise concerns about data oversight (Nature)
• Jeffrey Aronson: When I Use a Word . . . Retraction (BMJ Opinion)
• After retractions of two Covid-19 papers, scientists ask what went wrong (STAT)
• Will the pandemic permanently alter scientific publishing? (Nature)