Effectiveness of adverse effects search filters: drugs versus medical devices

Abstract

The study tested the performance of adverse effects search filters when searching for safety information on medical devices, procedures, and diagnostic tests in MEDLINE and Embase. The sensitivity of 3 filters was determined using a sample of 631 references from 131 rapid reviews related to the safety of health technologies. The references were divided into 2 sets by type of intervention: drugs and nondrug health technologies. Keyword and indexing analysis were performed on references from the nondrug testing set that 1 or more of the filters did not retrieve. For all 3 filters, sensitivity was lower for nondrug health technologies (ranging from 53%-87%) than for drugs (88%-93%) in both databases. When tested on the nondrug health technologies set, sensitivity was lower in Embase (ranging from 53%-81%) than in MEDLINE (67%-87%) for all filters. Of the nondrug records that 1 or more of the filters missed, 39% of the missed MEDLINE records and 18% of the missed Embase records did not contain any indexing terms related to adverse events. Analyzing the titles and abstracts of nondrug records that were missed by any 1 filter, the most commonly used keywords related to adverse effects were: risk, complications, mortality, contamination, hemorrhage, and failure. In this study, adverse effects filters were less effective at finding information about the safety of medical devices, procedures, and tests compared to information about the safety of drugs.

Headnote

Objective: The study tested the performance of adverse effects search filters when searching for safety information on medical devices, procedures, and diagnostic tests in MEDLINE and Embase.

Methods: The sensitivity of 3 filters was determined using a sample of 631 references from 131 rapid reviews related to the safety of health technologies. The references were divided into 2 sets by type of intervention: drugs and nondrug health technologies. Keyword and indexing analysis were performed on references from the nondrug testing set that 1 or more of the filters did not retrieve.

Results: For all 3 filters, sensitivity was lower for nondrug health technologies (ranging from 53%-87%) than for drugs (88%-93%) in both databases. When tested on the nondrug health technologies set, sensitivity was lower in Embase (ranging from 53%-81%) than in MEDLINE (67%-87%) for all filters. Of the nondrug records that 1 or more of the filters missed, 39% of the missed MEDLINE records and 18% of the missed Embase records did not contain any indexing terms related to adverse events. Analyzing the titles and abstracts of nondrug records that were missed by any 1 filter, the most commonly used keywords related to adverse effects were: risk, complications, mortality, contamination, hemorrhage, and failure.

Conclusions: In this study, adverse effects filters were less effective at finding information about the safety of medical devices, procedures, and tests compared to information about the safety of drugs.

Keywords: Information Storage and Retrieval, Controlled Vocabulary, Bibliographic Databases, Sensitivity and Specificity, Equipment and Supplies

Safety issues are a common thread across the wide range of health technologies, including drugs, medical devices, surgical procedures, and diagnostic tests. It is important that health professionals, researchers, and policy makers can access information on adverse effects for all of these different types of health technologies. While research has been done on designing optimal search strategies to retrieve information about adverse effects from bibliographic databases, this research has focused almost exclusively on adverse effects associated with pharmaceuticals [1-4]. It is unclear how well these filters perform for nondrug topics. In a 2014 case study of a systematic review on a device, Golder et al. found that the most effective terms to use when searching for adverse effects of a medical device differed from the most effective adverse effects terms when searching for pharmacological safety [5]. They highlighted the need for a search filter specifically targeted to the adverse effects of medical devices. Using the same systematic review as a case study, they also found that searching multiple sources was necessary to identify information about adverse effects for devices [6].

The objective of this study was to compare the performance between drug and nondrug health technologies of three adverse effects search filters. Additionally, this study attempted to identify indexing or keyword terms that might be useful in searching for nondrug safety topics.

METHOD

Filters tested

The authors selected filters for testing if they were developed within the past ten years and had translations for both MEDLINE and Embase via Ovid. Two published adverse effects filters were identified through a literature search: Golder et al. [4] and BMJ Clinical Evidence [7]. For testing purposes, Golder et al.'s most sensitive search strategy (which excludes the use of adverse effects terms specific to a particular technology) was used. A third filter developed for internal use by information specialists at the Canadian Agency for Drugs and Technologies in Health (CADTH) was also tested. Online only Appendix A provides details of the tested search filters.

Development of testing set

A testing set of references related to the adverse effects of drugs and nondrug health technologies was compiled to determine the sensitivity of the filters. The testing set included references from CADTH Rapid Response reports related to the safety of health technologies that were published between 2006 and 2012. Rapid Response reports are rapid evidence reviews with different levels of comprehensiveness; for this study, reviews with reference lists, a summary of abstracts, or a summary with critical appraisal were included. Online only Appendix B provides a flow diagram of the selection of reports and included references. Eligible Rapid Response reports were identified from searching the CADTH website using the following search string: safety OR harm OR harms OR ''adverse events'' OR ''adverse event'' OR ''adverse effects'' OR ''adverse effect'' OR ''side effect'' OR ''side effects.'' Two reviewers screened the results. Disagreements were resolved by consensus. Reports were then classified by topic category: drug, device, procedure, or diagnostic test. Reports were excluded if the report had no research question on safety, no safety studies were included, or if a safety filter was used in the original search. Out of 354 rapid reviews screened, 131 met the inclusion criteria. References from these reports were screened, and 631 articles related to safety were identified. If these articles were indexed as records in MEDLINE or Embase, they were included in the testing set. The testing set for MEDLINE included 362 records on drug topics and 223 records on other health technologies. The testing set for Embase included 356 records on drug topics and 216 records on other health technologies.

Unique record identifiers assigned by MEDLINE and Embase were used to isolate the testing set of relevant drug and nondrug records in the databases. The testing sets were combined using the Boolean operator ''AND'' with the adverse effects filters to test their sensitivity. Filter sensitivity was calculated by determining the number of records that a filter retrieved from the testing set divided by the total number of records in the testing set.

Keyword and indexing analysis

Records not retrieved by one or more of the filters in MEDLINE and Embase were downloaded into Reference Manager. The titles and abstracts for these records were extracted and analyzed using the word frequency software Text Analyzer [8]. References were manually screened to identify potential indexing terms relevant to the safety of medical devices, procedures, and diagnostic tests. Subject headings and subheadings were only counted once per record.

RESULTS

Filter sensitivity

All filters tested were less effective at retrieving relevant records from the nondrug health technologies testing set compared to the drug testing set (Table 1). On average, for all filters in both databases, the sensitivity for drugs was 91%, while the average sensitivity for nondrug health technologies was 69%. In MEDLINE, the decrease in sensitivity for nondrug interventions was 5% (BMJ Clinical Evidence [7]), 18% (Golder et al. [4]), and 21% (CADTH). This sensitivity gap between drug and nondrug topics was more pronounced in Embase for all 3 filters: 12% (BMJ Clinical Evidence), 35% (Golder et al.), and 37% (CADTH).

Keywords in missed records

Out of the 439 records in both MEDLINE and Embase nondrug testing sets, 119 unique records were not retrieved by 1 or more of the filters. Of these missed records, the most frequently used words in titles or abstracts that related to adverse effects were: ''risk'' (98 instances), ''complications'' (69 instances), ''mortality'' (23 instances), ''contamination'' (21 instances), ''hemorrhage'' (21 instances), ''failure'' (18 instances), and ''complication'' (18 instances). ''Blood loss'' (28 instances) was the most frequently used phrase related to safety.

Indexing terms in missed records

In MEDLINE, 76 records out of 223 in the nondrug testing set were missed by 1 or more of the filters. The most commonly used Medical Subject Headings (MeSH) in this set of missed records were ''Postoperative Complications'' (8 records), ''Risk Assessment'' (6 records), and ''Postoperative Pain'' (5 records). The subheading ''Complications'' was applied in 6 records. Thirty-nine percent of the missed records (30) in MEDLINE contained no indexing terms related to adverse effects.

In Embase, 1 or more filters missed 114 records out of 216 records in the nondrug testing set. The most frequent Emtree terms used in these missed records were ''Bleeding'' (17 records), ''Postoperative Pain'' (13 records), ''Postoperative Complication'' (12 records), and ''Risk Factor'' (12 records). The most frequently used safety-related subheading was ''Complication'' (38 records). Eighteen percent of the missed records (20 records) in Embase had no indexing terms related to adverse effects.

DISCUSSION

Search filters for adverse effects were found to be less effective at retrieving relevant information on medical devices, procedures, and diagnostic tests compared to drug topics in both MEDLINE and Embase. Filter sensitivity in Embase was lower than in MEDLINE for nondrug topics. The results of this study support Golder et al.'s findings that current search filters might not be adequate when searching for adverse effects of devices [5]. Differences in indexing (both subject headings and subheadings), variation in the language used to describe adverse effects, and types of safety issues that can occur between different health technologies could explain the lower sensitivity of adverse effects filters for nondrug topics in both databases.

Subject headings

It is worth exploring possible differences in controlled vocabulary between drug and nondrug health technologies when searching for adverse effects information. The BMJ Clinical Evidence and CADTH filters contain some subject headings that are specific to drugs, such as ''Drug Toxicity'' and ''Drug Hypersensitivity.'' The BMJ Clinical Evidence filter also includes some terms that apply mainly to procedures, such as ''Postoperative Complications'' and ''Intraoperative Complications,'' which could explain the BMJ Clinical Evidence filter 's higher sensitivity for nondrug interventions compared to the two other filters. Additionally, there are other potentially relevant subject headings for nondrug topics that are not included in any of the tested adverse effects filters. These include: in MeSH, ''Medical Device Recalls,'' ''Equipment Failure,'' and ''Equipment Contamination''; and in Emtree, ''Device Recall,'' ''Device Safety,'' and ''Medical Device Complication.''

It is important to note, however, that many MEDLINE records and some Embase records that one or more filters missed did not have any safetyrelated subject headings. Further, when safetyrelated headings were present, they were often specific to a particular device or procedure, such as ''Needlestick Injuries'' or ''Catheter Infection.''

Subheadings

The appropriateness of subheadings could also change, based on whether one is searching for drugs, devices, or procedures. In Embase, for example, the subheading ''Adverse Drug Reaction'' was most relevant to pharmaceuticals, whereas the subheading ''Complication'' was the most frequently applied in the set of missed nondrug records (38 of out 114 records). In MEDLINE, the subheading ''Complications'' appeared less frequently in the records that 1 or more of the filters missed (6 out of 76 records). All 3 filters included ''Complications'' as a subheading in their MEDLINE versions, but only the BMJ Clinical Evidence Embase filter included the corresponding ''Complication'' Emtree subheading. This might, in part, explain the lower sensitivity of the filters in Embase.

Keywords

Variations in the vocabulary of safety issues for different health technologies should be considered for keywords as well. The terms ''contamination,'' ''hemorrhage,'' and ''failure,'' for instance, might be more applicable to device and procedure safety than to drug safety. These terms were some of the most frequently found words in the titles and abstracts of the nondrug records that the safety filters missed. In contrast, terms such as ''toxicity'' or ''tolerability,'' which were included in the filters tested, might be more useful for drug topics.

Variation in nondrug health technologies

Given the variety in nondrug health technologies- from infusion pumps to X rays to dermal fillers-it is likely that the vocabulary used to describe safety issues is less standardized than for drugs. This dissimilarity means that any filter for adverse effects of devices, procedures, or diagnostic tests may need to be modified for each search by adding specific terminology based on the topic. In a search on intrauterine contraceptive device safety, for example, additional terms could include ''malposition'' or ''migration'' as keywords, as well as the Emtree heading ''Intrauterine Device Expulsion.'' A search on adverse effects of prenatal X-ray exposure may include the Embase heading ''Cancer Risk'' and the keyword ''rhabdomyosarcoma.'' Golder et al. also reported a wide variety of adverse effects terms found in studies included in a systematic review of the safety of recombinant human bone morphogenetic protein-2 for spinal fusion [5].

An additional complication is terminology that could be either an adverse effect or a symptom that a health technology is meant to treat. For example, ''hemorrhage'' as a keyword or heading may be useful for searches on surgical complications but would not be useful when searching on interventions designed to treat blood loss. Such ambiguous headings may not be useful in a standard filter; it may be more practical for searchers to consider such adverse effects headings on a topic-by-topic basis.

The results of this study support Golder et al.'s findings that different approaches may be required when searching for safety information on nondrug topics versus drug topics [5]. It highlights the need for tailored search strategies for adverse effects of nondrug health technologies that incorporate specific headings and keywords for medical devices, procedures, and diagnostic test safety.

Limitations

This study only tested the basic filters themselves and not additional approaches often used for safety searches. Golder et al., for example, recommended searching specific adverse effects based on topic area [2]. At CADTH, it is a common practice among information specialists to use the CADTH adverse filter only in combination with an observational filter so that randomized controlled trials and systematic reviews on a health technology would be searched without having a safety filter applied. Such approaches are likely to increase the sensitivity of searches, but it was not practical to measure them in this study because of the wide range of different health technologies included in the sample.

The test set was created for this study and might not represent the universe of the adverse events literature. There is no known representative sample of adverse events literature. Further, the sample included only a small number of results related to diagnostic tests, so it is especially unclear if the results are generalizable to this type of nondrug health technology.

This study only assessed whether relevant records were retrieved from the testing sets; it did not take into account precision-a measure of how many irrelevant records a filter retrieves-which is an important factor in filter performance. This study, therefore, cannot compare in a practical way the performance between filters. For example, a filter could retrieve all relevant drug and nondrug records, but if it also retrieves almost as many irrelevant records as searching without a filter, then it does not have much functional value. Future studies should take into account the precision of these filters.

ACKNOWLEDGMENTS

We acknowledge the assistance of Amanda Hodgson, David Kaunelis, Eftyhia Helis, and Kinneret Globerman in reviewing and providing feedback on the draft manuscript.

References

REFERENCES

1. Derry S, Loke YK, Aronson JK. Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials. BMC Med Res Methodol. 2001;1:7.

2. Golder S, McIntosh HM, Duffy S, Glanville J, Centre for Reviews and Dissemination and UK Cochrane Centre Search Filters Design Group. Developing efficient search strategies to identify reports of adverse effects in MEDLINE and EMBASE. Health Inf Libr J. 2006 Mar; 23(1):3-12.

3. Golder S, Loke YK. Sensitivity and precision of adverse effects search filters in MEDLINE and EMBASE: a case study of fractures with thiazolidinediones. Health Inf Libr J. 2012 Mar;29(1):28-38.

4. Golder S, Loke YK. The performance of adverse effects search filters in MEDLINE and EMBASE. Health Inf Libr J. 2012 Jun;29(2):141-51.

5. Golder S, Wright K, Rodgers M. Failure or success of search strategies to identify adverse effects of medical devices: a feasibility study using a systematic review. Syst Rev. 2014;3:113.

6. Golder S, Wright K, Rodgers M. The contribution of different information sources to identify adverse effects of a medical device: a case study using a systematic review of spinal fusion. Int J Technol Assess Health Care. 2014 Oct;30(4):423-9.

7. BMJ Clinical Evidence. Study design search filters [Internet]. London, UK: BMJ Publishing Group; 20 Sep 2012 [cited 10 Jul 2015]. ,http://clinicalevidence.bmj.com/ x/set/static/ebm/learn/665076.html..

8. Text analyzer [Internet]. Online-Utility.org; 2009 [cited 10 Jul 2015]. ,http://www.online-utility.org/text/analyzer. jsp..