Covidence and Rayyan
Purpose: Systematic review software.
URLs: https://www.covidence.org/ and http://rayyan.qcri.org/.
Cost: Free.
Bottom line: Two new software products are making a splash in the world of systematic reviews, Covidence and Rayyan. Both have been developed from within the systematic review community, by and for users, on a not-for-profit basis. Rayyan is free for anyone, and Covidence is free for authors of Cochrane Reviews. Both are user-friendly and work well for title and abstract screening.
Purpose
The purpose of systematic review software is to facilitate the process of screening and data extraction from many studies according to prespecified criteria of the review. Where I work (McMaster University, Department of Anesthesia, Michael G. DeGroote Institute of Pain Research and Care, ICRP), large and complex systematic reviews are our major focus. We therefore rely on DistillerSR (Ottawa, Canada; http://systematic-review.net/), the excellent systematic review software developed by Evidence Partners. But I also run literature searches for other systematic reviews outside of the scope of the ICRP for various teams, and some review teams just don’t have the funding to purchase licences for DistillerSR. In supporting these review teams, I have explored Covidence and Rayyan and found they both work beautifully to support the tasks associated with title and abstract screening and study selection. Covidence and Rayyan are user-friendly and a real improvement over other low-cost title and abstract screening approaches such as sorting references into groups in Endnote, entering codes in Reference Manager, making a table in a Word document, or printing them out and marking them in pen. If your team has a large set of records to screen for eligibility, then do yourselves a favour and learn to use Rayyan or Covidence.
Product description and cost
Rayyan is completely free and offers reviewers the capability to screen titles and abstracts offline using the mobile app, for even greater cost efficiency. Its unique machine-learning function lets Rayyan make suggestions for labels based on your pattern of selection, and it “learns” from your include/exclude decisions, giving a five-star rating to those articles you are most likely to include. This is a great way to offset the effects of reviewer fatigue, and the “similarity graph” visual display is just plain cool. Rayyan comes from Qatar Computing Research Institute, HBKU, a member of Qatar Foundation and is available at http://rayyan.qcri.org/.
Covidence has a free trial option (one review with two reviewers) and is free for use in Cochrane Reviews. In addition to support for title and abstract screening, it offers tools for quality assessment and data extraction that are optimized for Cochrane (or Cochrane-style) intervention reviews (wherein trial results of a specific treatment for a specific clinical condition are pooled). Covidence is produced in collaboration with the Cochrane Collaboration; Australia’s Monash University, Alfred Hospital and, National ICT; England’s University College; and Argentina’s Instituto de Effectividad Clinica y Sanitaria. It is available at https://www.covidence.org/.
Intended audience
For this product review, I am focussing on individuals who would like to support the initial stages of a systematic review. The team librarian is typically involved in designing and running the literature search, but further delivering the search results in a format that facilitates title and abstract screening adds value and supports the review team, giving them more time to focus on data abstraction and analysis. If you are not familiar with the task of title and abstract screening for systematic reviews, a great way to learn more about it is on the Cochrane Crowd site, where anyone can screen records for inclusion in the Cochrane Library’s Central database of Controlled Trials (http://crowd.cochrane.org/index.html). Both Covidence and Rayyan operate in a similar way to the Cochrane Crowd interface: a bibliographic record pops up on your screen with various keywords highlighted, you read the abstract of the record, then make a judgement about whether or not it meets your inclusion criteria. Click on the button that reflects your judgement (yes, no, or unsure) and move along to the next record. Because the interfaces for screening titles and abstracts are all so similar, if anyone on your team has ever screened abstracts for eligibility for a systematic review, chances are they will be able to figure out how to screen records in Covidence or Rayyan in about two minutes. If you are all newbies, take the time to review the introductory materials and plan for about 30 minutes to get the hang of it.
Platform, usability, and compatibility
JCHLA Product Review guidelines suggest that info on platform and usability be addressed for computer programs. Believe me, these tools are user-friendly. But if you need more detailed, specific technical information, some helpful resources to consult as you decide which software to use for your systematic review project are the Software for Systematic Reviewing page on the HLWiki (http://hlwiki.slais.ubc.ca/index.php/Systematic_reviewing_software) and the Systematic Review Toolbox (http://systematicreviewtools.com/). For compatibility, the key to success with all of these tools is that you need to map or parse the bibliographic information from the references into the correct fields in the database, and when your review team adds information to the references, you want to be able to export that information in a sensible and stable format. You might say that the entire purpose of Covidence, Rayyan, or systematic review software in general is to be more user-friendly than a basic spreadsheet, but the issues of compatibility are the same—your columns and rows need to line up.
Special features, strengths, and weaknesses and comparison with other products
Jessica Babineau published a great review of how to use Covidence in JCHLA Vol. 35 No. 2 (https://ejournals.library.ualberta.ca/index.php/jchla/article/view/22892/17064) and while the basic how-to-do it steps are the same, there have been significant improvements to the interface since 2014, most significantly the excellent Knowledge Base of static help files available at http://support.covidence.org/help_center where they have short video clips showing how to do everything. I had never used Covidence, and in 2015 I had an urgent request from an international team with six reviewers. They were all up and running (screening articles for selection) on Covidence within a day. Other improvements to the Covidence interface include:
| • | identification of duplicate references (Figure 1), |
| • | option to display reference numbers, |
| • | option to enter inclusion and exclusion criteria text on the screening form, and |
| • | ability to export PRISMA flow-chart data |
When you import references into Covidence, you can choose which level of the review to import them to (i.e., screen, full-text review, included, excluded, or irrelevant), and reports can also be exported from every level as a text or .csv file. With these reports you can keep records of the screening process, saving reports at each step of the consensus project when work is performed in duplicate (this is important if you want to calculate agreement with, for example, the kappa statistic). The only feature still on the wish list at Covidence is the capacity to bulk import PDFs (coming soon!).
Fig. 1. Screen shot showing duplicate identification in Covidence.
An interesting thing about Covidence that you should keep in mind is that it randomly “serves up” articles to be reviewed, rather than assigning sets of articles to each reviewer. As long as you are not used to assigning sets of articles to reviewers, this isn’t a problem. It prevents the “clumping” of articles within sets, e.g., by the lead author’s surname. On the other hand, if you are used to assigning a set of references to each reviewer, you will need to communicate with your team about the Covidence approach, otherwise you might find one eager beaver screening more references than you expected! Another thing that caused us a moment of dismay in Covidence was that we lost the Record Numbers when we exported our records using the Endnote format. A work-around for this was to export into.csv format instead.
Covidence has a mobile app, but you have to be online or at least using your mobile phone to use it, and this is where Rayyan really offers the “killer app” that will make it the software of choice for many review teams. With Rayyan, you can screen offline and then synchronize your work with the server the next time you login. I really like this feature, because screening titles and abstracts is a relaxing task ideally suited to such internet-free occasions as plane trips or on the bus. I ran a test review on Rayyan with two reviewers, and they both found it easy to use. One was a total newbie and he figured it out in no time, and the other was an experienced reviewer who noted that there is a handy Review chat function (Figure 2) which is a great place to record your inclusion criteria and make comments about the review.
Fig. 2. Rayyan Review Chat screenshot showing reviewer feedback.
When you upload or import your references to Rayyan, they first appear in an “undecided” folder. As reviewers make their decisions, the references move to included, excluded, or conflict folders. By exporting the articles from each of these folders as the review progresses, it is possible to keep track of consensus agreement and generate the values to calculate your kappa, if required. From the Rayyan Review Workbench, with the Blind ON, each reviewer proceeds independently. If you turn the Blind to OFF, each reviewer’s includes (green) or excludes (red) are seen (Figure 3).
Fig. 3. Rayyan Review Workbench screenshot.
Records in Rayyan are assigned a seven-digit system identification number upon import. I’m not sure what happens to the record numbers for records exported from Endnote or other bibliographic software (they seem to disappear, but perhaps asking for this field to be displayed is already on the “Help us decide what to add next” list). Tech support at Rayyan is very helpful and responsive. For my test review in Rayyan, my reviewers did not screen enough records to trigger the machine-learning five-star rating process. You need at least 50 records with ≥10% included for this feature, which is a very helpful tool for reviews that have thousands of records. For example, if after you finish your screening, you find that certain five-star records should happen to be excluded, it will be easy to take a second look at them and verify whether or not they belong in the trash. Maybe someone was asleep at the switch, or maybe it is a false positive. For example an American College of Physicians Journal Club evidence summary of a randomized controlled trial (RCT) or an editorial about an RCT might rank high in similarity if the topic of RCT meets inclusion criteria, but it may be excluded from a systematic review. On the other hand a five-star RCT on your topic should be included. I explored the Similarity Graph, a unique Rayyan feature related to the five-star ranking algorithm, and think this picture of a set of references is lovely (Figure 4). Khabsa et al. [1] described what the Similarity Graph represents.
Fig. 4. Screenshot of Rayyan similarity graph.
Once the studies have been selected the real work of the review begins: data extraction and analysis. At the MLA–CHLA conference I took the opportunity to converse with the Covidence representative on the topic of data extraction, and she used a bicycle metaphor to illustrate the trade-off that exists in software design, between simplicity or ease of use and the complexity to robustly manage complicated, customized queries. For data extraction, Covidence is like one of those bicycle-share vehicles found in cities, convenient and sturdy (useful for intervention reviews, with middle-of-the road outcomes.) However, it will not get you to the velodrome like a racing bike or up the Niagara escarpment like a mountain bike (complex or unusual data extraction with various or unusual outcomes.) Although they can’t compete with DistillerSR for data extraction, for the tasks of managing citations for title and abstract screening and study selection, Rayyan and Covidence can be a great help and leave your team with greater energy to devote to the tasks that follow.
Rachel Couban
Research Coordinator
DeGroote Institute for Pain Research and Care
McMaster University
1280 Main St West
Hamilton, ON L8S 4K1
Email: [email protected]
1. Khabsa M, Elmagarmid AK, Ilyas IF, Hammady H, Ouzzani M. Learning to identify relevant studies for systematic reviews using random forest and external information. Mach Learn. 2016;102(3):465–482. doi:10.1007/s10994-015-5535-7 Publisher Full Text
© 2016. This work is published under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.