Content area
Abstract
Objectives
To collate, review, and comment upon publishers’ response to integrity concerns.
Study Design and Setting
We conducted a narrative review of publications reporting the responses of publishers to concerns about the integrity of research published in their journals. We also drew upon extensive personal experience and a new analysis of publisher responses to integrity concerns about 172 clinical trial publications by a single research group 5 years after the concerns were raised simultaneously with affected publishers.
Results
Existing evidence reports that slow, incomplete, and opaque responses from publishers to integrity concerns are common, in both clinical and preclinical disciplines. When we raised very similar concerns about a large set of journal articles simultaneously with publishers, times to resolution varied markedly, and outcomes ranged from no editorial action to all papers retracted.
Conclusion
Publishers' responses to notification of concerns about the integrity of publications in their journals are markedly inconsistent, both in their timing and the nature of their editorial decisions. The reasons for these inconsistencies are unknown but could be addressed by a collaborative and transparent process involving publisher integrity staff and academics with expertise in publication integrity. Understanding the reasons for the disparate outcomes is likely to facilitate improvements which will enhance the trustworthiness of the biomedical literature.
Plain Language Summary
Existing evidence reports that publishers are slow to assess concerns about the reliability of research publications, and their assessments produce markedly inconsistent outcomes. Our finding of widely disparate outcomes of publisher assessments of overlapping concerns about 172 clinical trials by a single research group reinforces this point. Improving the timeliness, transparency, and systematicity of publisher assessments is likely to enhance the reliability of published research.
Full text
What is new?
Key findings
• Outcomes of publishers' assessments of the same concerns about publication integrity vary considerably in both timing and nature.
• It is not possible to determine the reason(s) for these inconsistencies because publishers do not provide itemized responses to detailed concerns.
What this adds to what was known?
• The integrity of the biomedical literature can be compromised by the processes currently undertaken with the intention of protecting it.
What is the implication and what should change now?
• Publishers should develop consistent, systematic, timely, and transparent processes for assessing publication integrity, including drawing upon independent expertise.
Publication integrity is the honest, comprehensive, and accurate reporting of research. Its importance is indisputable. Untrustworthy publications harm health, derail science, and waste resources. Assuring publication integrity is a collective responsibility involving the entire scientific community, but ultimately is the responsibility of publishers, who oversee what is published and what is corrected. Publishers and their journals are the first point of contact when readers identify integrity concerns and must coordinate assessments and attempt to resolve those concerns. Assessment and maintenance of publication integrity is a quality-control process which ideally would be efficiently applied and produce consistent outcomes. As guidance from the Committee on Publication Ethics (COPE) is readily available to publishers to assist their integrity assessments, it might be expected to facilitate consistency of outcomes.
Timeliness and consistency of integrity assessments are important because the consequences of delays in notifying readers of concerns about the integrity of publications are considerable. Potentially untrustworthy clinical publications can be incorporated into narrative reviews, systematic reviews and clinical guidelines, thereby influencing patient care. If that happens, and the research in question is then determined to be untrustworthy, it is no small matter to correct the influential documents it taints. For example, we judged that 45 of 88 clinical guidelines and systematic reviews that included 27 subsequently retracted clinical trial publications by Yoshihiro Sato and colleagues would have different conclusions after removal of the retracted work [ 1]. Contacting the authors and editors of the tainted reviews elicited a response from only 51% and a correction for only 10% 1 year later. Potentially untrustworthy psreclinical research influences the direction and focus of research that may inform subsequent clinical research endeavors. Flawed laboratory and clinical research publications about Alzheimer's disease likely hindered progress in the field by focusing attention of researchers on incorrect hypotheses and results [ 2]. The opportunity costs are large when high profile publications are flawed and corrective editorial action delayed.
1 Accumulating evidence for inconsistent outcomes of integrity assessments
To our knowledge, there are no published reports demonstrating timely assessments and consistent outcomes after integrity concerns are raised. To the contrary, there is a growing body of published evidence that efficiency and consistency are not being achieved [ 3]. For example, after a range of integrity concerns were raised with publishers about 891 clinical publications in women's health, resolution was achieved for only 263 (30%). For the publications with an editorial decision, there was a median of 38 months from the time of notification to resolution. Among publishers, the proportions of completed assessments ranged from 0% to 78%, and there was marked variation in the decisions rendered [ 4]. Six years after the fabricated research by Yoshitaki Fujii and colleagues was identified, 20% of affected publications were unretracted and additional retractions only occurred after further notifications of journals by readers [ 5]. The Retraction Watch website (a project of the Center for Scientific Integrity) regularly features descriptions of slow or absent assessments of integrity concerns and/or surprising outcomes.
More robust examination of publishers' responses to integrity concerns is possible when the same group of concerns is notified to a group of publishers and their journals. Here, the available evidence also reports markedly inconsistent outcomes. After journals were notified of large amounts of identical data in 39 clinical publications (9 duplicate, three triplicate, and three quadruplicate) first-authored by the same researcher, the concerns about 25 of the publications had not been resolved 2 years later. Of those which were resolved, four were retracted and 10 were not retracted [ 6]. When overlapping concerns were raised about the integrity of 10 publications by a single research group in the clinical pain literature [ 7], four were retracted and six had no action taken. In a set of 36 preclinical publications which featured duplicate reporting of results, authorship transgressions, and data discrepancies, resolution of the concerns was achieved only for 16 (44%) after 1 year. There was considerable variation in the decisions reached – nine no action, three correction, four retraction [ 8]. In laboratory science, publisher responses to notification of a shared error — the use of a purportedly inert control reagent that in fact had biological activity — were similarly inconsistent. For the 31 affected publications, the final outcomes were retraction in 14, expression of concern in 4, author correction in 7, and no action in 6 [ 9].
2 New evidence
Our recent experience permits a stronger evaluation of the consistency and timeliness of outcomes among publishers, because we notified the same concerns about the integrity of 172 clinical trial publications from 2011 to 2019 by a research group with a common author (Zatollah Asemi) to all affected journals and publishers with a single email in July 2019. The concerns, which were pervasive and applied both to the entire body of work and to specific publications within it, are published [ 10], as are the methods we used for the integrity assessments [ 10–12]. We analyzed summary baseline data and participant withdrawal information from all publications and compared the observed distributions for each with the expected distributions derived from a large set of trustworthy trial publications [ 10, 11]. We compared trial registration information for each publication with that reported in the linked publication(s) [ 12]. We used the 11 domain REAPPRAISED (Research governance, Ethics, Authorship, Productivity, Plagiarism, Research conduct, Analyses and methods, Image manipulation, Statistics and data, Errors, Data duplication and reporting) publication integrity checklist [ 13] to itemize concerns about many individual publications. We assessed publisher responses and outcomes 5 years later. Our analyses focused on the nine publishers which each had >five affected publications (range 8–31). Collectively, they were responsible for 148 (86%) of the affected publications. Eight of the nine publishers are members of the COPE; the other (Thieme) follows policies “based on the recommendations of COPE” [ 14]. For each publisher, email correspondence indicated that the assessments were overseen by its research integrity staff.
The
Figure
Notification of journal readers about the existence of integrity concerns also varied markedly among publishers, both in timing and extent. The Figure (lower panel) shows the time from receipt of concerns to the issuing of any editorial notice. In all cases, the first editorial notice was an expression of concern. The first publisher to issue an editorial notice (Springer Nature) did so 7 months after receipt of the concerns document. In contrast, no journal published by Wiley issued an editorial notice until 43 months after receipt of the concerns document. Two publishers (Elsevier and Wolters Kluwer) have not issued any editorial notices, more than 60 months after receiving the concerns document. Among the seven publishers that have issued editorial notices, four did so en masse over a few days. The others issued editorial notices over 6 months, 7 months, and 18 months.
The nomenclature and visibility of the expressions of concerns also varied. Five publishers (Taylor & Francis, Cambridge University Press, Oxford Academic Press, Karger, and Wiley) issued notices labeled and published as expressions of concern, 1 publisher (Thieme) issued notices labeled as “notes of concern”, and 1 publisher (Springer Nature) issued notices labeled as “editors' notes”, followed by an expression of concern for two publications. All but one of the notices labeled as expressions of concern are listed on PubMed, but none of the editors' notes or notes of concern is, 2–4 years after being issued.
3 Implications and improvements
A growing body of evidence indicates that publishers' responses to notification of concerns about the integrity of publications in their journals are markedly inconsistent, both in the timing and the nature of editorial decisions. Median times to editorial decisions typically exceed 2 years. Equally disconcerting is the observation that, when faced with a common set of integrity concerns about publications from the same researchers, some publishers decide to retract much or all of the research and others take no action. While some variability in time to resolution might reasonably be attributed to publisher case load it is difficult to understand differences of several years between publishers in time to resolution. If publisher case load is a significant contributing factor to delays of several years, it implies that some publishers are underresourcing their integrity efforts.
Given that the time to resolution of integrity concerns, whatever the outcome, is often years rather than months, it is surprising that there is such marked variation in notification of readers that assessment of integrity concerns is being undertaken. COPE guidance recommends the publication of an expression of concern if “an investigation is underway but a judgment will not be available for a considerable time” [ 15], although it does not define what constitutes a considerable time. The National Information Standards Organization considers that “to ensure effective communication of expressions of concern, the same principles apply as for retractions” [ 16]. One of those principles is that of wide dissemination, including publication in the online and/or print issue of the journal. In the analysis reported herein, two publishers chose to only express concern about their publications online, a practice which necessarily reduces the visibility of the notice to readers. If the point of notices of concern is to alert readers that an assessment of integrity is underway the notices should be made as visible to readers as possible. That way, readers can determine whether to trust the research in question while assessments are undertaken. As things stand, readers of the work by Asemi and colleagues that is published in affected Taylor & Francis and Wiley journals will know it is untrustworthy or potentially untrustworthy, readers of affected Thieme and most Springer Nature journals will only learn it is potentially untrustworthy if they visit journal websites, and readers of affected Elsevier and Wolters Kluwer journals will not learn that it is potentially untrustworthy.
It is not possible currently to understand why slow and inconsistent publisher responses to integrity concerns occur, since those responses lack transparency and detail. Regular posts on the Retraction Watch website indicate that this is a general problem. Our experience is that, despite notification of concerns using an itemized checklist, we never receive a point by point response to the concerns, and retraction notices do not reflect the scope and nature of the concerns raised [ 17]. It is unusual to receive a copy of the authors' responses to the concerns, or to receive a copy of the report of an independent review if one is obtained. In contrast, journals routinely request point by point responses by authors to peer reviewers during manuscript submission. Confidence in integrity assessments would be improved if journals treated integrity concerns in a similar fashion to prepublication reviews, with the same commitment to resourcing, intent, efficiency, and transparency. It is unusual for a publisher to provide reasons for prolonged and/or inconsistent responses to those who raise integrity concerns. There may be multiple factors at play, including heavy publisher workload, publisher underresourcing, high publishing staff turnover, substandard communication systems, inadequate record keeping, uncertainties about who has primary responsibility for components of the assessment process, and concerns about litigation by disgruntled authors. Each of these factors has been invoked by publishers in their correspondence with us about integrity concerns. It would greatly assist the field if publishers were to audit and report their integrity cases, including the concerns raised, timelines, the assessments undertaken, the reasons for the final decision, and the reasons for delays. This could be done without compromising confidentiality.
A notable aspect of assessments undertaken by publishers is that guidance from COPE is very frequently invoked as pivotal in their management of publication integrity. A typical communication from a publisher is that it is “investigating the concerns in line with guidance from COPE”. However, the disparities in timelines and outcomes that are observed after integrity concerns are raised suggest either that COPE guidance is not being properly followed by publishers or that the guidelines themselves are ambiguous or lack key components. We think COPE guidance could be improved in a number of ways to assist its members in resolving publication integrity concerns [ 18]. Numbering COPE guidance items would allow specific reference to the exact guidance being followed. Revised guidance should prioritize what matters most to journal readers, publication integrity, ahead of the question of researcher behavior (“misconduct”) [ 13, 19]. This would reduce the need for, and reliance upon, institutional investigations, which focus primarily on researcher behavior and are often slow, conflicted, incomplete [ 20], and/or opaque [ 21]. COPE could recommend greater scrutiny by publishers of key study documents and information, including raw data, at the time of manuscript submission [ 22]. A systematic examination of individual patient data for clinical trial reports submitted to one journal found that 25%–45% contained false data [ 23]. COPE could recommend early, visible, and standardized notification of readers of the existence and nature of important integrity concerns. It could recommend timelines for the resolution of integrity concerns; a UK parliamentary committee recommended 2 months [ 24], but 6 months may be more realistic. COPE could recommend systematic and methodology-driven assessments of integrity concerns, drawing upon the wealth of academic expertise in these areas. It could recommend that publishers report how integrity assessments are undertaken, in sufficient detail to permit journal readers to understand how the final decision was reached. If such changes were made, and publishers, whether COPE members or not, followed the revised guidance, it might be possible to greatly improve the system that is responsible for assuring publication integrity [ 25].
CRediT authorship contribution statement
Andrew Grey: Writing – review & editing, Writing – original draft, Investigation, Formal analysis, Data curation, Conceptualization. Alison Avenell: Writing – review & editing, Investigation, Data curation, Conceptualization. Alan Gaby: Writing – review & editing, Data curation. Mark J. Bolland: Writing – review & editing, Investigation, Data curation, Conceptualization.
Declaration of competing interest
There are no competing interests for any other author.
© 2025 The Author(s)