COVIDENCE AND RAYYAN Covidence

Abstract

Kellermeyer et al reviews two systematic reviews tools, the Covidence and Rayyan.

COVIDENCE AND RAYYAN Covidence. Covidence, Level 4, 549 St Kilda Road, Melbourne Victoria, 3004, Australia; [email protected]; https://www.covidence.org; trial version: free with 2 reviewers; single review: $240 with unlimited reviewers; package: $445 includes 3 reviews with unlimited reviewers.

Rayyan. Rayyan, HBKU Research Complex, P.O. Box 5825, Doha, Qatar; https://rayyan.qcri.org; free.

INTRODUCTION

Increasingly important in health care settings, systematic reviews (SRs) aim to identify, evaluate, and summarize the relevant studies of a health-related issue. Librarians respond to the need for SRs with expert search services, project management, and training, making them well positioned to help teams establish best practices during the review process.

To minimize bias and present reliable evidence, SRs adhere to a design based on structured and reproducible methods, which include steps to search for relevant studies, extract data, assess the quality of the data, and then analyze and present the results. Several groups, such as Cochrane and the Institute of Medicine, have set forth guidelines for conducting systematic reviews [1, 2]. Additionally, Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) formatting is commonly adopted to summarize and report SR findings.

Numerous tools are available that have been designed to help with one or more steps in the SR process. The aim of this review is to compare two SR tools, Covidence and Rayyan, and examine the ways in which they support SR methodology and reporting standards. Covidence is free for Cochrane authors and fee-based for others. It was developed specifically to guide reviewers through a prescribed SR workflow. Rayyan is a completely free tool developed to expedite the SR process by easing citation sharing and allowing comparison of decisions to include or exclude.

Before starting a review project, participants should be clear about the degree to which and the means by which they will adhere to best practices. While methodological quality is distinct from reporting quality, one cannot evaluate how well the SR was conducted with incomplete documentation [3]. Both are needed for an effective summary of the evidence. Transparency, reproducibility, and publisher-defined requirements should all inform the selection of an SR tool.

COVIDENCE

Covidence was developed by an Australian not-for-profit company. In 2015, Cochrane initiated a partnership with Covidence that made it the standard production platform for Cochrane reviews. While an instance of Covidence (a single review) is free for Cochrane authors, others receive 1 free trial with a maximum of 2 reviewers and then face fees. One instance costs $240, which includes an unlimited number of reviewers. Pricing schedules vary and include options for institutional subscriptions and bulk purchasing. Covidence mirrors the multiphase review process, including data extraction, directly in its design. Citations neatly progress through each stage based on votes received. At every stage, reviewers can explicitly assign voting roles, including tie breaking, while maintaining blinding, which helps to minimize bias.

RAYYAN

Rayyan was developed through the Qatar Computing Research Institute, funded by the Qatar Foundation, a nonprofit that supports education, science, research, and community development initiatives in Qatar. It is completely web-based, with offline compatibility through its app. Users are able to initiate and/or participate in an unlimited number of reviews. As opposed to Covidence, Rayyan does not easily mirror the multiphase citation review process and is really only designed to aid with the reference screening. It takes a minimalist approach, placing more of the logistical and workflow burden on the users themselves.

COMPARISON

Table 1 compares the two products. In essence, Covidence protects the integrity of the systematic review process. It lacks flexibility by design in order to minimize the chance that reviewers could do anything to compromise the review process or data reporting. It has built-in capabilities to handle the screening process in a way that reduces duplicate efforts on behalf of the reviewers. For example, if 100 articles need a vote by 2 reviewers and there is a team of 3 reviewers, Covidence will track include/exclude decisions as they are made, removing an article from the review queue if it has met the requirement of receiving two votes. In this way, the review workload can be spread more easily amongst the team. In contrast, Rayyan's reviewer blinding function can only be turned on or off. If the same team of 3 wants a blinded review of those 100 articles, the articles would need to be manually split up or each reviewer would need to vote on all articles.

Resolving conflicts is easier in Covidence, which has an integrated conflict resolution workflow that allows a designated person to be a tie-breaker. Though blinding is possible during initial voting in Rayyan, conflicts can only be resolved by unblinding the decisions, having reviewers discuss, and manually changing votes. A chat feature is available to aid in this process.

Once articles pass into the fulltext phase, Covidence requires that reviewers add a single reason when choosing to exclude an article. If different reviewers choose different reasons for exclusion, Covidence will classify it as a conflict that needs to be resolved. This gives each article a single exclude reason, providing tidy data for the final exclude table needed for SR reporting.

In contrast, there is no full-text phase designated in Rayyan's interface. The best practice for mimicking this phase would be to export the citations that were included in the first pass and import them back into a new Rayyan review with the initial decisions stripped from them. After that, full-text portable document format files (PDFs) could be uploaded individually, and reviewers could proceed as in the first pass. Rayyan allows reviewers to skip adding an exclude reason or to add multiple reasons. While resolving all conflicts at the full-text phase is possible through Rayyan, a workflow would need to be established prior to beginning. Data necessary for the exclude table would need to be manually extracted.

Finally, Covidence offers tools and a workflow for all data extraction that would be necessary for the final SR reporting. It produces a PRISMA flow diagram and is able to export additional information to RevMan, a software program that can produce a meta-analysis and graphic representation of the data. Rayyan does not support any additional phases of the SR workflow past the screening. Covidence is well-suited for more rigorous SRs, where methodology must be adhered to and documented at each stage. Though workflow and features can feel clunky, Rayyan offers a nice structure for the initial screening process and works as a viable upgrade from a workflow that uses only EndNote and/or Excel.

Sidebar

Liz Kellermeyer, [email protected], https://orcid.org/0000-0001-6115-4343, Biomedical Research Librarian, Library and Knowledge Services, National Jewish Health, Denver, CO

Ben Harnke, ben.harnke@ucdenver. edu, Education and Reference Librarian, Health Sciences Library, University of Colorado Anschutz Medical Campus, Aurora, CO

Shandra Knight, knights@njhealth. org, https://orcid.org/0000-0002-4404-3833, Director, Library and Knowledge Services, National Jewish Health, Denver, CO

References

REFERENCES

1. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions [Internet]. Version 5.1.0. London, England, UK: Cochrane Collaboration; 2011 [cited 24 Aug 2018]. <http://handbook.cochrane.org>.

2. Institute of Medicine. Finding what works in health care: standards for systematic reviews. Washington, DC: The National Academies Press; 2011. DOI: http://dx.doi.org/10.17226/13059.

3. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, Ramsay T, Bai A, Shukla VK, Grimshaw JM. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One. 2007;2(12):e1350. http://dx.doi.org/10.1371/iournal.pon e.0001350.