Competing with a pandemic: Trends in research design in a time of Covid-19

Abstract

Introduction

During the Covid-19 pandemic, major journals have published a significant number of Covid-19 related articles in a short period of time. While this is necessary to combat the worldwide pandemic, it may have trade-offs with respect to publishing research from other disciplines.

Objectives

To assess differences in published research design before and after the Covid-19 pandemic.

Methods

We performed a cross-sectional review of all 322 full-length research studies published between October 1, 2019 and April 30, 2020 in three major medical journals. We compared the number of randomized controlled trials (RCTs) and studies with a control group before and after January 31, 2020, when Covid-19 began garnering international attention.

Results

The number of full-length research studies per issue was not statistically different before and after the Covid-19 pandemic (from 3.7 to 3.5 per issue, p = 0.17). Compared to before January 31, 2020, 0.7 fewer non-Covid-19 studies per issue were published versus after January 31, 2020 (p<0.001), a change that was offset by Covid-19 studies. Among non-Covid-19 studies, 0.9 fewer studies with a control group per issue were published after January 31, 2020, with RCTs contributing to nearly all the decline (p<0.001, p = 0.001, respectively). In the same timeframe, non-Covid-19 studies without a control group and non-Covid-19 studies without randomization experienced relatively small changes that did not meet our threshold for statistical significance (increases of 0.1 and 0.1 per issue, p = 0.80, p = 0.88, respectively).

Limitations

Using a simple heuristic for assessing research design and lack of generalizability to the general medical literature.

Conclusions

In summary, the increase in Covid-19 studies coincided with a decrease of mostly non-Covid-19 RCTs.

About the Authors:

Shelly X. Bian

Roles Conceptualization, Data curation, Methodology, Supervision, Validation, Writing – original draft, Writing – review & editing

Affiliation: Department of Radiation Oncology, University of Southern California, Los Angeles, California, United States of America

ORCID logo http://orcid.org/0000-0003-0599-019X

Eugene Lin

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliation: Division of Nephrology and Hypertension, Department of Medicine, University of Southern California, Los Angeles, California, United States of America

ORCID logo http://orcid.org/0000-0002-3559-5134

Introduction

The Covid-19 pandemic has severely disrupted medical research, from shutting down laboratory facilities, to delaying clinical trials, to halting funding [1, 2]. Simultaneously, the volume of Covid-19 research has proliferated as experts from all disciplines seek to combat the new threat [3, 4]. In the months following the onset of Covid-19, there has been an unprecedented increase in the number of registered clinical trials, preprints, and publications related to Covid-19 [5].

As of May, 2020, ClinicalTrials.gov listed over 1000 Covid-19 studies. Although some will advance our clinical knowledge, many are small, poorly designed, redundant, and unlikely to prove clinically useful [6]. Meanwhile the peer review process for Covid-19 studies has accelerated substantially. An analysis of 14 medical journals found a 50% reduction in average turnaround time from submission to publication for Covid-19 related research. This study, however, did not include several of the most influential journals including The Journal of the American Medical Association, The Lancet, and The New England Journal of Medicine [7]. The proliferation of Covid-19 literature has permeated into non-peer reviewed preprint websites, such as BioRxiv and MedRxiv. As of May 7, MedRxiv and BioRxiv had close to 2800 Covid-19 related articles combined. This has led to several editorials expressing concern over the quality of Covid-19 related research [8, 9].

Many people look to most influential medical journals as having the highest standards for publication. The shifting of our research priorities is appropriate and necessary to address the pandemic. However, this diversion may also have unintended consequences on research in other disciplines. To evaluate whether the Covid-19 pandemic had negative effects on published research in other disciplines, we studied characteristics of Covid-19 and non-Covid-19 related research in three of the most cited medical journals: The Journal of the American Medical Association, The Lancet, and The New England Journal of Medicine. Although prior studies have evaluated the proliferation, quality, and design of Covid-19 related publications, none to our knowledge have addressed the pandemic’s effect on the study design of non-Covid-19 medical research. In this cross-sectional study, we investigated whether the design of published non-Covid-19 research changed in three of the highest impact medical journals aimed at a predominantly clinical audience after the onset of the Covid-19 pandemic.

Methods

Identifying research studies

We reviewed all full-length research studies published between October 1, 2019 and April 30, 2020 in three of the highest impact medical journals, The New England Journal of Medicine (NEJM), the Journal of the American Medical Association (JAMA), and The Lancet. These included “Original Articles” from NEJM (including “Brief Reports” and “Special Reports”), “Original Research” from JAMA (including “Preliminary Communications” and “Special Communications”), and “Research Articles” from The Lancet.

Many Covid-19 studies are case series published as short research studies and letters to the editor. In a sensitivity analysis, we additionally included short research studies published in letter format: “Correspondences” from NEJM and The Lancet and “Research Letters” from JAMA. The Correspondence sections from the NEJM and The Lancet have a broad range and include letters addressing previous articles, primary research, and opinions. From these, we manually identified all letters with a primary data collection effort (including case series and translational research) or secondary analysis of already collected data. We included all of JAMA’s Research Letters. We excluded sections dedicated for case reports (e.g., the “Case Records from the Massachusetts General Hospital in The New England Journal of Medicine” or “Brief Reports” from JAMA) but included case reports or case series published as short research studies or letters to the editor. Our publication selection process is outlined in a Prisma flow diagram in Fig 1.

[Figure omitted. See PDF.]

Fig 1. PRISMA flow diagram.

https://doi.org/10.1371/journal.pone.0238831.g001

We assigned each study an organ system (S1 Table in S1 Appendix) and determined whether it was (i) Covid-19 related, (ii) a study with a control group, and (iii) a randomized controlled trial (RCT). We defined Covid-19-related research as having any Covid-19 related words in the title of the article including “Covid-19”, “novel coronavirus”, “2019-nCoV”, “Coronavirus disease 2019”, or SARS-CoV-2”. We defined a control group as whether the investigators performed a hypothesis-driven analysis, comparing two or more groups. Studies describing trends over time were considered uncontrolled unless the investigators tested a specific hypothesis, such as analyzing the putative effect of a policy intervention. Both authors reviewed all studies, adjudicating conflicts by consensus (see S1 Appendix for additional formal designations).

Statistical analyses

We compared study characteristics before and after the end of January 2020, when Covid-19 began garnering international attention. We defined the pre-Covid-19 era as January 31, 2020 and prior, and the post-Covid-19 era as February 1, 2020 and after. We chose this cutoff because the last week of January, 2020 marked the public announcement of the first cases of Covid-19 outside of China. Simultaneously, the World Health Organization (WHO) began warning other countries of the virus’ global spread and the importance of formulating strategic plans for virus containment around this time. Furthermore, the first Covid-19 publication in the three medical journals of interest was published on February 15, 2020 in The Lancet.

We first plotted the number of published Original Research studies, studies that were Covid-19 related, studies with and without a control group, and studies that were RCTs over time. We performed this descriptive analysis by assigning each issue to a given week, anchoring to the Monday of that week. We computed the average number of studies per week before and after the end of January 2020 to visually demonstrate changes in before and after Covid-19.

Subsequently, we assessed per issue differences in study type before and after the end of January 2020. For each of the three journals examined, issues are published weekly with the rare exception of an omitted week. We used the Mann-Whitney-Wilcoxan test to assess whether the average number of Covid-19 related studies, studies with and without a control group, and studies that were RCTs per issue was different before and after January 31, 2020. We performed analyses using the number of studies per issue instead of the proportion of studies per issue because counts reflect the zero-sum nature of publications and provide meaningful information on the volume of studies published. For instance, a large decrease in the total number of published studies might not change the proportion of published RCTs but would likely decrease the number of published RCTs. Because the total number of published studies per issue did not change substantially over time, this distinction was unlikely material to our results.

Our primary focus was on all full-length research studies and on the non-Covid-19 related subgroup of studies. In a sensitivity analyses, we expanded our sample to all research studies including those in letter format.

To assess changes in publication by organ system, we aggregated the total number of full-length research articles into pre-Covid-19 and post-Covid-19 cohorts. We did this because of the scarcity of some organ systems at an issue level. Because the pre-Covid-19 study period had more months, we plotted the percent of published full-length research studies before and after the onset of Covid-19. For each organ system, we compared the significance of the change in proportion before and after Covid-19 using Fisher’s exact test.

We acknowledge that we did not pre-register our study with a predetermined statistical protocol. Additionally, we performed a large number of statistical tests. Therefore, we performed a conservative Bonferroni correction [10] to adjust the threshold for rejecting the null hypothesis. Because we performed 21 tests in our primary analysis, we required a p-value of 0.0024 when determining whether a difference was statistically significant.

Results

We identified 322 total full-length research studies, 188 before and 134 after the end of January, 2020. The distribution of full-length research articles by journal for each month is detailed in Table 1.

[Figure omitted. See PDF.]

Table 1. Distribution of research article type by journal.

https://doi.org/10.1371/journal.pone.0238831.t001

The total number of full-length research studies did not statistically differ before and after Covid-19 (from 3.7 to 3.5 per issue, p = 0.17). After January 2020, non-Covid-19 full-length studies decreased by 0.7 per issue on average (p<0.001), which was offset by Covid-19 full-length studies. The total number of full-length studies with a control group decreased by 0.8 per issue on average (p<0.001) and the number of RCTs decreased by 0.9 per issue (p = 0.001). Fig 2 shows changes in full-length articles by week, which usually includes 1 issue from each journal.

[Figure omitted. See PDF.]

Fig 2. Changes in full-length research studies published before and after Covid-19.

Figure includes all full-length research studies in counts per week. A) Covid-19 versus non-Covid-19 studies, B) controlled versus uncontrolled studies, C) RCTs versus non-RCTs. P-values estimated using a two-sample t-test. The gray vertical lines denote January 31, 2020, the boundary between the pre- and post-Covid-19 eras. Solid horizontal lines denote average values of the solid dots and dashed horizontal lines denote average values of the hollow dots.

https://doi.org/10.1371/journal.pone.0238831.g002

Among non-Covid-19 full-length studies, the number of studies with a control group decreased by 0.9 per issue on average with RCTs contributing to nearly all the decline (p<0.001, p = 0.001, respectively). In the same timeframe, non-Covid-19 full-length studies without a control group and non-Covid-19 full-length studies without randomization did not statistically differ (p = 0.80, 0.88, respectively). Fig 3 shows weekly changes in full-length non-Covid-19 articles.

[Figure omitted. See PDF.]

Fig 3. Changes in full-length non-Covid-19 research studies published before and after Covid-19.

Figure includes all non-Covid-19 full-length research studies in counts per week. A) controlled versus uncontrolled studies B) RCTs versus non-RCTs. P-values estimated using a two-sample t-test. The gray vertical lines denote January 31, 2020, the boundary between the pre- and post-Covid-19 eras. Solid horizontal lines denote average values of the solid dots and dashed horizontal lines denote average values of the hollow dots.

https://doi.org/10.1371/journal.pone.0238831.g003

Cardiology and obstetrics/gynecology had the largest declines in full-length published studies, from 23% to 13% (9% decline, p = 0.04) and from 4% to 0% (4% decline, p = 0.02, respectively. Meanwhile infectious diseases had the largest increase, from 9% to 28% (19% increase, p<0.001) (Fig 4, Table 2).

[Figure omitted. See PDF.]

Fig 4. Distribution of full-length research studies pre- and post-Covid-19 by specialty.

P-values were estimated using Fisher’s Exact Test. (an asterisk, *, denotes p < 0.0025). January 31, 2020 is the boundary between the pre- and post-Covid-19 eras.

https://doi.org/10.1371/journal.pone.0238831.g004

[Figure omitted. See PDF.]

Table 2. Change in full-length research articles pre- and post-Covid-19 by specialty.

https://doi.org/10.1371/journal.pone.0238831.t002

In a sensitivity analysis, we included an additional 104 short research studies (57 pre- and 47 post-Covid-19) typically published in Letter format. The sensitivity analysis included a total of 426 research studies, 425 pre- and 181 post-Covid-19 and was not materially different from our primary analysis.

The total number of studies overall did not statistically differ before and after Covid-19 (from 4.9 to 4.8 per issue, p = 0.95). After January 31, 2020, non-Covid-19 studies decreased by 1.2 per issue on average (p<0.001), which was offset by Covid-19 studies. We observed a decrease in number of studies pre- and post-Covid-19 with a control group of 0.9 per issue on average (p = 0.001), which was almost entirely explained by a decrease in RCTs (p = 0.002). Fig 5 shows weekly changes in all articles.

[Figure omitted. See PDF.]

Fig 5. Changes in all research studies published before and after Covid-19.

Figure includes full-length research studies as well as research studies in Letter form in counts per week. A) Covid-19 versus non-Covid-19 studies, B) controlled versus uncontrolled studies, C) RCTs versus non-RCTs. P-values estimated using a two-sample t-test. The gray vertical lines denote January 31, 2020, the boundary between the pre- and post-Covid-19 eras. Solid horizontal lines denote average values of the solid dots and dashed horizontal lines denote average values of the hollow dots.

https://doi.org/10.1371/journal.pone.0238831.g005

Among non-Covid-19 studies, the number of studies with a control group decreased by 0.9 per issue on average with RCTs contributing to nearly all the decline (p<0.001, p = 0.002, respectively). In the same timeframe, non-Covid-19 studies without a control group and non-Covid-19 studies without randomization did not statistically differ (p = 0.27, 0.26, respectively). Fig 6 shows weekly changes in all non-Covid-19 articles.

[Figure omitted. See PDF.]

Fig 6. Changes in all non-Covid-19 research studies published before and after Covid-19.

Figure includes all non-Covid-19 full-length research studies as well as research studies in Letter form. A) controlled versus uncontrolled studies B) RCTs versus non-RCTs. P-values estimated using a two-sample t-test.

https://doi.org/10.1371/journal.pone.0238831.g006

In contrast, none of the Covid-19 studies (20 full-length and 21 in letter format) were RCTs, and only one had a control group. Table 3 shows changes in articles per issue in all articles as well as full-length articles before and after January 31, 2020.

[Figure omitted. See PDF.]

Table 3. Changes in articles per issue before and after January 31, 2020, the boundary between the defined pre- and post-Covid eras.

https://doi.org/10.1371/journal.pone.0238831.t003

Discussion

We found that in three of the highest impact medical journals, the increase in Covid-19-related research studies coincided with a concomitant decrease in non-Covid-related RCTs. Unsurprisingly, most Covid-19 studies published in the first three months of the pandemic comprised case reports and case series. No Covid-19 related RCT’s were published from February 1, to April 30, 2020. We observed a decreasing trend in cardiology and obstetrics/gynecology studies post-Covid-19, though the decrease did not meet our threshold for statistical significance.

Prioritizing Covid-19 research is critical to combating the pandemic [1]. The medical community needs and expects the quick dissemination of Covid-19 research. However, hasty research may result in suboptimal study designs, though some have argued for the need to balance scientific rigor for speed [11]. Similarly, Kim et. al. recently argued in the Annals of Internal Medicine has stated that “Given the urgency of the situation, some limitations… may be acceptable, including the small sample size, use of an unvalidated surrogate end point, and lack of randomization or blinding” [8]. Still, researchers and publishers must use caution. In the last month alone, four high-profile Covid-19 articles have been retracted from the Annals of Internal Medicine, The Lancet, and NEJM due to inadequate scientific rigor [12–15]. We similarly observed a large increase in Covid-19 studies that did not have a control group, mostly case reports and series. As Covid-19 researchers have time to apply more rigorous methods to their studies, RCTs and studies with control groups will undoubtedly become more widespread. In the meantime, researchers and journal editors will need to balance the trade-off between accommodating the rapid dissemination of information with Covid-19 research that do not report a control group.

However, we were concerned to observe substitutions occurring at the expense of non-Covid-19 studies with a control group, with substitutions occurring almost entirely at the expense of RCTs. In comparison, we observed minimal changes among non-Covid-19 studies without a control group. If we project these changes over a course of 6 months (a conservative time frame for the Covid-19 pandemic), non-Covid-19 studies with a control group could decrease by 60 and non-Covid-19 RCTs by 70 in these three medical journals combined. Notably, in the field of obstetrics/gynecology, zero studies were published in the post-Covid-19 era despite constituting 4% of full-length publications prior to the outbreak. The decline in non-Covid-19 publications with a control group and especially RCTs likely has multiple reasons. Publication of already accepted studies may be postponed in lieu of emerging Covid-19 research. Journals in the JAMA Network have received 53% more submissions in the first quarter of 2020 than in the first quarter of 2019 [16]. The flood of submissions and the demand for Covid-19 related work presents publishers and editors with the dilemma of accepting fewer non-Covid-19 publications, quality notwithstanding.

Reviewers and editors might also perceive non-Covid-19 research as overall less relevant given the pandemic. The WHO, national disease organizations, and media have Covid-19 in the forefront of public health. Since January, 2020, COVID-19 papers have been downloaded more than 150 million times, according to the International Association of Scientific, Technical and Medical Publishers [17]. Major publications have a responsibility to assess the interest of their readership and adjust their publication profile accordingly. Finally, authors might delay submissions until after the pandemic subsides because of unavoidable disruptions in their research or a perception that non-Covid-19 research is viewed less favorably. Many laboratories and non-Covid-19 clinical trials have closed due to social distancing [18]. Additionally, resources and priorities may have shifted away from non-Covid-19 research to support Covid-19 studies [19].

Unfortunately, these rejections and delays contribute to a growing backlog of unpublished non-Covid-19 research, hindering future dissemination. The backlog will likely worsen as non-Covid-19 research resumes with long-term negative consequences to published research [9]. Disruptions in medical research attributable to Covid-19 could thus mirror public health experts’ concerns about delays in non-Covid-19 medical care [20].

Limitations of our study include using a simple heuristic for assessing study design (e.g., RCT versus not, controlled versus uncontrolled), not accounting for contemporaneous idiosyncratic events (e.g., vaping), and not observing submitted but unpublished work. Additionally, we did not produce a written protocol and preplanned analysis prior to data collection. We also limited our sample to three select high impact journals, and our results do not necessarily generalize to the entire medical literature. We were unable to estimate changes pre- and post-Covid-19 by organ system with precision, likely owing to small sample size. Finally, we used the 3 months preceding the start of Covid-19 to assess the pre-Covid-19 baseline for the three journals of interest. However this surrogate may not be a suitable counterfactual for these journals in the absence of Covid-19. Thus, our findings should be corroborated in a wider sample of journals and time periods.

Substitutions of non-Covid-19 research are inevitable. To alleviate delays in publication, editors could consider establishing dedicated channels for Covid-19 studies that do not compete with non-Covid-19 studies.

Supporting information

[Figure omitted. See PDF.]

S1 Appendix. Additional details for “Competing with a Pandemic: Research Quality in a Time of Covid-19”.

https://doi.org/10.1371/journal.pone.0238831.s001

(DOCX)

Acknowledgments

We would like to thank Gemma Lin for her review of the manuscript.

Citation: Bian SX, Lin E (2020) Competing with a pandemic: Trends in research design in a time of Covid-19. PLoS ONE 15(9): e0238831. https://doi.org/10.1371/journal.pone.0238831

References

1. Chong S-A, Capps BJ, Subramaniam M, Voo TC, Campbell AV. Clinical Research in Times of Pandemics. Public Health Ethics. 2010 Apr 1;3(1):35–8.

2. Widener A. Journal publishers promote flexibility during COVID-19 pandemic. Chem Eng News (C EN) [Internet]. 2020 Apr 6 [cited 2020 Jul 27];98(13):14–14. Available from: https://pubs.acs.org/doi/abs/10.1021/cen-09813-polcon2

3. Doroshow D, Podolsky S, Barr J. Biomedical Research in Times of Emergency: Lessons From History. Ann Intern Med [Internet]. 2020 May 7 [cited 2020 May 18];M20-2076. Available from: https://www.acpjournals.org/doi/10.7326/M20-2076

4. Di Girolamo N, Reynders RM. Characteristics of scientific articles on COVID-19 published during the initial three months of the pandemic: a meta-epidemiological study. MedRxiv Prepr [Internet]. [cited 2020 May 18]; Available from: https://www.medrxiv.org/content/10.1101/2020.04.20.20073130v1

5. Brainard J. Scientists are drowning in COVID-19 papers. Can new tools keep them afloat? Science (80-). 2020 May 13;

6. Glasziou PP, Sanders S, Hoffmann T. Waste in covid-19 research [Internet]. Vol. 369, The BMJ. BMJ Publishing Group; 2020 [cited 2020 Jul 6]. Available from: https://www.bmj.com/content/369/bmj.m1847

7. Horbach SPJM. Pandemic Publishing: Medical journals drastically speed up their publication process for Covid-19. [cited 2020 May 18]; Available from: https://doi.org/10.1101/2020.04.18.045963

8. Alexander PE, Debono VB, Mammen MJ, Iorio A, Aryal K, Deng D, et al. COVID-19 coronavirus research has overall low methodological quality thus far: case in point for chloroquine/hydroxychloroquine [Internet]. Vol. 123, Journal of Clinical Epidemiology. Elsevier USA; 2020. Available from: https://pubmed.ncbi.nlm.nih.gov/32330521/

9. London AJ, Kimmelman J. Against pandemic research exceptionalism. Science (80-). 2020 Apr 23;368(6490):476–7.

10. Mittelhammer RC, Judge GG, Miller DJ. Econometric Foundations. Cambridge University Press; 2000. 73–74 p.

11. Kupferschmidt K. WHO launches global megatrial of the four most promising coronavirus treatments. Science (80-). 2020 Mar 22;

12. Mehra MR, Desai SS, Kuy S, Henry TD, Patel AN. Retraction: Cardiovascular Disease, Drug Therapy, and Mortality in Covid-19. N Engl J Med. N Engl J Med [Internet]. 2020 Jun 25 [cited 2020 Jul 27];382(26):2582–2582. Available from: https://www.nejm.org/doi/full/10.1056/NEJMc2021225?query=recirc_curatedRelated_article

13. Zeng Y, Zhen Y. RETRACTED: Chinese medical staff request international medical assistance in fighting against COVID-19. Lancet Glob Heal [Internet]. 2020 Feb [cited 2020 Jul 27];0(0). Available from: https://doi.org/10.1016/S2214-109X

14. Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED:Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet [Internet]. 2020 [cited 2020 Jul 27];0(0). Available from: https://doi.org/10.1016/S0140-6736

15. Bae S, Kim M-C, Kim JY, Cha H-H, Lim JS, Jung J, et al. Notice of Retraction: Effectiveness of Surgical and Cotton Masks in Blocking SARS-CoV-2. Ann Intern Med [Internet]. 2020 Jun 2 [cited 2020 Jul 27]; Available from: https://www.acpjournals.org/doi/abs/10.7326/L20-0745

16. Kwon D. How swamped preprint servers are blocking bad coronavirus research. Vol. 581, Nature. NLM (Medline); 2020. p. 130–1.

17. COVID-19 Research Downloaded More Than 150 Million Times, Reports STM [Internet]. International Association of Scientific, Technical and Medical Publishers. [cited 2020 Jul 27]. Available from: https://www.prnewswire.co.uk/news-releases/covid-19-research-downloaded-more-than-150-million-times-reports-stm-824020806.html

18. Servick K. Updated: Labs go quiet as researchers brace for long-term coronavirus disruptions. Science (80-). 2020 Mar 16;

19. Ledford H. Coronavirus shuts down trials of drugs for multiple other diseases. Vol. 580, Nature. NLM (Medline); 2020. p. 15–6.

20. Wong, Laura E, Hawkins, Jessica E, Langness S, Murrell, Karen L, Iris, Patricia, Sammann A. Where Are All the Patients? Addressing Covid-19 Fear to Encourage Sick Patients to Seek Emergency Care. NEJM Catal. 2020;