Failure or success of electronic search strategies to identify adverse effects data

Abstract

[...]electronic searches based on specific adverse effect-related terms could miss nearly a quarter of relevant papers. [...]in a systematic review that assessed risk of cancer arising as an adverse drug reaction, only cancer-related indexing terms qualified.\n Generic adverse effects terms in the title and abstract in any of the 3 databases, generic and specific indexing terms or subheadings for adverse effects in EMBASE, and adverse effects subheadings in MEDLINE retrieved the highest proportion of records.

INTRODUCTION

One of the key difficulties in conducting specific searches for adverse effects stems from the absence of adverse effects terms in the title, abstract, and indexing of relevant articles. In 2001, Derry et al. evaluated articles that reported adverse effects data from clinical trials and found that about 23% of such articles had no adverse effects terms (either generic or specific) in the title, abstract, or indexing of records in MEDLINE or EMBASE [1]. Hence, electronic searches based on specific adverse effect-related terms could miss nearly a quarter of relevant papers. Current guidance on systematic reviews of adverse effects emphasizes the use of nonspecific searches (without relying on adverse effects terms) as well as the need to check full-text versions of retrieved articles to ensure a complete search [2]. Unfortunately, implementation of such guidance is onerous and time consuming compared to running more specific adverse effects-based searches.

However, methodological developments in the past decade might have changed the situation since Derry et al.'s study [1]. In 2003, ten new recommendations about the reporting of harms were added to the Consolidated Standards of Reporting Trials (CONSORT), including a recommendation that if the study collected data on harms, the title or abstract should state this [3]. Additionally, the Cochrane Adverse Effects Methods Group <http://www .aemg.cochrane.org>, which was formed in 2007, as well as many other authors, called for improved reporting of adverse effects [4-32]. However, it is unclear if these developments have had a meaningful impact on the prevalence of adverse effects terms in the title, abstract, or indexing of relevant adverse effects papers.

This study collected a large, diverse, and contemporaneous cohort of articles with adverse effects data so that the presence or absence of specific adverse effects terms could be assessed [I].

METHODS

Selection of cohort of studies

The first step involved collecting a cohort of papers that were known to report data on the frequency of adverse effects but that had not been retrieved using specific adverse effects search terms.

Identification of systematic reviews of adverse effects through hand searching of the Database of Abstracts of Reviews of Effects. The primary studies were identified from papers included in systematic reviews of adverse effects that were indexed as full abstracts in the Database of Abstracts of Reviews of Effects (DARE), one of the databases available from the Centre for Reviews and Dissemination website <http://www.crd .york.ac.uk/crdweb/ >. A systematic review was considered eligible for inclusion if it was published between 2007 and 2010 and:

A. Adverse drug effects were the primary outcome.

B. Specific adverse effects search terms or specified/ named adverse effects had not been used by the review authors to locate articles for the review. This created a cohort in which relevant articles had not already been chosen because of presence of specific adverse effects terms.

C. The search included either hand searching or reference checking in addition to database searches (this is consistent with the study by Deny et al. 2001 [I]).

Selection of relevant primary studies with adverse effects data. The included references in each systematic review were checked for papers published in English after the year 2000, so that a contemporaneous study cohort could be obtained. Non-English-language papers were excluded because of the difficulty of obtaining valid matches for adverse effects terms in different languages. Full-text articles were checked to confirm the presence of adverse effects data that had been used in the systematic review. The papers were then de-duplicated to remove copies of papers that had been included in more than one systematic review.

Analysis

Availability in selected databases. The first stage of the analysis checked whether each paper was listed and indexed in MEDLINE, EMBASE, or Science Citation Index (SCI). MEDLINE and EMBASE are the most commonly searched databases in systematic reviews of adverse effects [33, 34] and were used in the study by Deny et al. [I]. SCI is also commonly used [33, 34], and a systematic review of thiazodinediones-related fractures (by the authors but not yet published) suggested that this database contained more relevant references than other databases, including MEDLINE and EMBASE.

Adverse effects terms in the database records. For each database, the available papers were sorted according to the following criteria:

1. The authors mentioned terms synonymous with "adverse effects" in the title or abstract (thus potentially enabling the paper to be found in an electronic search). Examples of synonyms are "adverse events," "side effects," "tolerated," and "unwanted effects." This is in line with terms accepted by Deny et al. 2001 [I].

2. The authors mentioned terms describing specific relevant adverse effects (such as "headache" or "cancer") in the title or abstract. The terms were considered relevant based on the adverse effects included in the systematic review. For example, for a systematic review on cancer as an adverse effect, only cancer-related terms were considered relevant. This part of the analysis was only conducted on included studies from reviews for a specific, named adverse effect.

3. The papers had been indexed in MEDLINE or EMBASE (using subject headings or subheadings) or allocated keywords in SCI with relevant terms for concepts related to adverse effects (thus potentially enabling the paper to be found in an electronic search). Relevant indexing terms were those that could be considered synonymous with "adverse effects" and would have been accepted by Deny et al. 2001 [I]. Examples of included indexing terms were "drug toxicity/" and "side effects/" and examples of included subheadings were "adverse effects (ae)" or "adverse drug reaction (ae)."

4. The papers had been indexed in MEDLINE or EMBASE or allocated keywords in SCI terms describing a specific adverse effect or adverse effects. This part of the analysis was only conducted on included studies from reviews where the focus was on specific, named adverse effects, and only indexing terms relating to these particular adverse effects qualified. For example, in a systematic review that assessed risk of cancer arising as an adverse drug reaction, only cancer-related indexing terms qualified.

RESULTS

Systematic reviews

Twenty-six systematic reviews met the inclusion criteria [35-6O]. Half were concerned with the safety profile for an intervention [35, 36, 41, 44-49, 52-54, 57], whereas the other half were limited to a named, specific adverse effect with an intervention [37-40, 42, 43, 50, 51, 55, 56, 58-6O].

Primary studies

Two hundred forty-two papers were eligible for use in the analysis. The majority of the papers were randomized controlled trials (RCTs) (89%, 216/242); however, there were also 13 case series, 5 chart reviews, 3 case reports, 3 cohort studies, 1 non-RCT study, and 1 uncontrolled study. Two hundred thirty-one of the 242 references were indexed in Ovid MEDLINE, 222 in EMBASE, and 238 in SCI.

Adverse effects terms

Searching with generic or specific, named adverse effects terms anywhere in the title, abstract, or indexing would have identified 89% of all relevant references in EMBASE, 80% in MEDLINE, and 70% in SCI (Table 1). Generic adverse effects terms in the title and abstract in any of the 3 databases, generic and specific indexing terms or subheadings for adverse effects in EMBASE, and adverse effects subheadings in MEDLINE retrieved the highest proportion of records.

Comparison with Deny et al. 2001

Overall, a combined search using terms in the title, abstract, or indexing in both MEDLINE and EMBASE would have failed to retrieve 19 (all of which were RCTs) of the 233 papers (8%) across all 26 systematic reviews. This is much lower than the 23% of papers that would have been missed with a similar search approach in Deny et al. 2001 [I].

DISCUSSION

It is reassuring to note that the prevalence of adverse effects terms in the title, abstract, or indexing (in MEDLINE and EMBASE for articles that are known to contain adverse effects data) has increased compared to previous findings [I]. In the past, specific searches for studies that contained adverse effects data have been hindered by the frequent absence of adverse effects terms. However, this study's findings indicate that reviewers can cautiously choose to use more focused search filters or specific adverse effects terms, rather than face the arduous task of broad nonspecific searches, followed by evaluation of full-text articles.

The frequency of specific adverse effects indexing terms and generic adverse effects indexing terms was much higher in EMBASE compared to MEDLINE or SCI and reflects the general practice in EMBASE of assigning more indexing terms to records. There was a lack of relevant keywords for adverse effects in the records from SCI. This is not surprising given that these terms are assigned either by the author when the paper is submitted for publication or from words or phrases that frequently appear in the titles of the references cited by the paper. This means that the adverse effect would need to be a key aspect of the paper or in the titles of references for that paper to appear as keywords. In addition, not all records in SCI contain keywords. For example, meeting abstracts tend not to include any keywords as the authors of abstracts generally do not have to provide keywords and abstracts generally do not include references. MEDLINE and EMBASE, on the other hand, have controlled vocabularies, and indexing is a manual process based on the full text of the article.

CONCLUSIONS

Adverse effects terms are increasingly prevalent in the title, abstract, and indexing terms of articles that contain adverse effects data in MEDLINE and EMBASE.

Source of funding

This research was undertaken by Su Golder as part of a Medical Research Council (MRC) fellowship. The views expressed in this presentation are those of the authors and not necessarily those of the MRC.

ACKNOWLEDGMENTS

We thank Professor Lesley Stewart, Centre for Reviews and Dissemination, for comments on an earlier draft.

References

REFERENCES

1. Deny S, Loke YK, Aronson JK. Incomplete evidence: the inadequacy of databases in tracing published adverse drug reactions in clinical trials. BMC Med Res Methodol. 2001;1(7).

2. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions 5.1.0 [updated Mar 2011]. The Cochrane Collaboration; 2011.

3. Ioannidis JP, Evans SJ, Gotzsche PC, O'Neill RT, Altaian DG, Schulz K, Moher D, Consort G. Better reporting of harms in randomized trials: an extension of the CONSORT statement. Ann Intern Med. 2004 Nov 16;141(10): 781-8.

4. Haidich A, Birtsou C, Dardavessis T, Tirodimos I, Arvanitidou M. The quality of safety reporting in trials is still suboptimal: survey of major general medical journals. J Clin Epidemiol. 201 1;64(2): 124-35.

5. Yazici Y. Safety reporting in randomized clinical trials: a need for improvement. Bull NYU Hosp Jt Dis. 2009;67(2): 209-10.

6. Yiannakopoulou E, Damianou C, Bamia C, Papdopulos JS. Adequacy of safety data reporting in randomized clinical trials comparing coxibs with non selective NSAIDs. Epitheorese Klinikes Farmakologias Kai Farmakokinetikes. 2005; 23(l):42-3.

7. Papanikolaou PN, Churchill R, Wahlbeck K, Ioannidis JP. Safety reporting in randomized trials of mental health interventions. Am J Psychiatry. 2004;161(9):1692-7.

8. Ioannidis JPA, Lau J. Improving safety reporting from randomised trials. Drug Saf. 2002;25(2):77-84.

9. Neidig JL, Koletar SL. Safety reporting in clinical trials. JAMA. 2001 Apr 25;285(16):2077-8.

10. Kohl KS, Bonhoeffer J, Steering Committee of the Brighton Collaboration. Safety reporting in clinical trials. JAMA. 2001 Apr 25;285(16):2076-7; author reply: 7-8.

11. Ioannidis JP, Lau J. Completeness of safety reporting in randomized trials: an evaluation of 7 medical areas. JAMA. 2001 Jan 24-31;285(4):437-43.

12. Chowers MY, Gottesman BS, Leibovici L, Pielmeier U, Andreassen S, Paul M. Reporting of adverse events in randomized controlled trials of highly active antiretroviral therapy: systematic review. J Antimicrob Chemother. 2009 Aug;64(2):239-50.

13. Yazici Y. Some concerns about adverse event reporting in randomized clinical trials. Bull NYU Hosp Jt Dis. 2008;66(2):143-5.

14. Cheng CW, Bian ZX, Li YP, Moher D, Wu TX, Dagenais S, Li J, Li TQ. Transparently reporting adverse effects of traditional Chinese medicine interventions in randomized controlled trials. Zhong Xi Yi Jie He Xue Bao/J Chin Integr Med. 2008 Sep;6(9):881-6.

15. Bauer KA, Hammerman S, Rapoport B, Lacouture ME. Completeness in the reporting of dermatologie adverse drug reactions associated with monoclonal antibody epidermal growth factor receptor inhibitors in phase II and ?? colorectal cancer clinical trials. Clin Colorectal Cancer. 2008 Sep;7(5):309-14.

16. Woodworth T, Fürst DE, Alten R, Bingham C, Yocum D, Sloan V, Tsuji W, Stevens R, Fries J, Witter J, Johnson K, Lassere M, Brooks P. Standardizing assessment and reporting of adverse effects in rheumatology clinical trials II: the Rheumatology Common Toxicity Criteria v.2.0. J Rheumatol. 2007 Jun;34(6):1401-14.

17. Nuovo J, Sather C. Reporting adverse events in randomized controlled trials. Pharmacoepidemiol Drug Saf. 2007 Mar;16(3):349-51.

18. Extermann M. More thoughts on the reporting of adverse events in cancer clinical trials. J Clin Oncol. 2007 Mar 1;25(7):918; author reply: 9.

19. Toenders WGM. Better reporting of adverse effects in clinical trials [Dutch]. Nederlands Tijdschrift voor Geneeskunde. 2006 Dec 23;150(51):2838.

20. Burfeind DB. Streamlining adverse event reporting in trials. Dermatol Nurs. 2006 Aug;18(4):381-2.

21. Anderson SJ. Some thoughts on the reporting of adverse events in phase ? cancer clinical trials [comment]. J Clin Oncol. 2006 Aug 20;24(24):3821-2.

22. Sather C, Nuovo J. Reporting methods of adverse events in randomized controlled trials [abstract]. International Congress on Peer Review and Biomedicai Publication: Chicago, IL, 16-18 Sep 2005. p. 54.

23. Lassere MND, Johnson KR, Woodworth TG, Fürst DE, Fries JF, Kirwan JR, Tugwell PS, Day RO, Brooks PM. Challenges and progress in adverse event ascertainment and reporting in clinical trials. J Rheumatol. 2005 Oct;32(10):2030-2.

24. Trotti A, Bentzen SM. The need for adverse effects reporting standards in oncology clinical trials. J Clin Oncol. 2004;22(l):19-22.

25. Liauw WS, Day RO. Adverse event reporting in clinical trials: room for improvement. Med J Aust. 2003 Oct 20;179(8):426-8.

26. Martin PJ, Antin JH, Weisdorf DJ, Patón V, Horowitz MM. Reporting of adverse event data in hematopoietic stem cell transplantation clinical trials involving investigational new drugs or devices: a report from the William Guy Forbeck Foundation 2001 focus meeting on clinical trials in hematopoietic stem cell transplantation. Biol Blood Marrow Transplant. 2002;8(6):295-302.

27. Loke YKDS. Lack of detail and uniformity in the reporting of adverse drug reactions in randomized controlled trials: important implications for systematic reviews 9th Annual Cochrane Colloquium Abstracts; Lyon, France, Oct 2001.

28. Loke YK, Deny S. Reporting of adverse drug reactions in randomised controlled trials: a systematic survey. BMC Clin Pharmacol. 2001;1(3).

29. Ethgen M, Boutron I, Steg PG, Roy C, Ravaud P. Reporting of harm in randomized controlled trials evaluating stents for percutaneous coronary intervention. Trials [Internet]. 2009;10:29 [cited 9 Mar 2011]. <http://www .trialsjournal.eom/content/10/l/29/>.

30. Ethgen M, Boutron I, Baron G, Giraudeau B, Sibilia J, Ravaud P. Reporting of harm in randomized, controlled trials of nonpharmacologic treatment for rheumatic disease. Ann Intern Med. 2005;143(l):20-5.

31. Ethgen M, Boutron I, Baron G, Giraudeau B, Sibilia J, Ravaud P. Reporting of harm in randomized controlled trials evaluating pharmacological and nonpharmacological treatments [abstract]. XIII Cochrane Colloquium; Melbourne, Australia; 22-26 Oct 2005. p. 97.

32. Papanikolaou P, Churchill R, Wahlbeck K, Ioannidis J. Reporting of harms in randomized trials of mental health interventions [abstract]. 12th Cochrane Colloquium; Bridging the Gaps: Ottawa, ON, Canada; 2-6 Oct 2004. p. 60-1.

33. Golder S, Loke Y, Mclntosh HM. Room for improvement? a survey of the methods used in systematic reviews of adverse effects. BMC Med Res Methodol. 2006;6:3.

34. Golder S, Loke Y, Mclntosh HM. Poor reporting and inadequate searches were apparent in systematic reviews of adverse effects. J Clin Epidemiol. 2008;61:440-8.

35. Agbabiaka T, Pittler M, Wider B, Ernst E. Serenoa repens (saw palmetto): a systematic review of adverse events. Drug Saf. 2009;32(8):637-47.

36. Albavera-Hernández C, Rodríguez J, Idrovo A. Safety of botulinum toxin type A among children with spasticity secondary to cerebral palsy: a systematic review of randomized clinical trials. Clin Rehabil. 2009;23(5):394-407.

37. Alghamdi AA, Moussa F, Fremes SE. Does the use of preoperative aspirin increase the risk of bleeding in patients undergoing coronary artery bypass grafting surgery? systematic review and meta-analysis. J Card Surg. 2007; 22(3):247-56.

38. Berlie H, Kalus J, Jaber L. Thiazolidinediones and the risk of edema: a meta-analysis. Diabetes Res Clin Pract. 2007;76(2):279-89.

39. Bonovas S, Sitaras N. Does pravastatin promote cancer in elderly patients? a meta-analysis. CMAJ. 2007;176(5):649-54.

40. Cao Y, Liu L, Liao C, Tan A, Gao F. Meta-analysis of incidence and risk of hypokalemia with cetuximab-based therapy for advanced cancer. Cancer Chemother Pharmacol. 2010;66(l):37-42.

41. Chavez-Tapia N, Tellez- Avila F, Bed ogni G, Crocè L, Masutti F, Tiribelli C. Systematic review and meta-analysis on the adverse events of rimonabant treatment: considerations for its potential use in hepatology. BMC Gastroenterol. 2009;9(75).

42. Chen L, Ashcroft D. Risk of myocardial infarction associated with selective COX-2 inhibitors: meta-analysis of randomised controlled trials. Pharmacoepidemiol Drug Saf. 2007;16(7):762-72.

43. Correli C. Weight gain and metabolic effects of mood stabilizers and antipsychotics in pediatrie bipolar disorder: a systematic review and pooled analysis of short-term trials. J Am Acad Child Adolesc Psychiatry. 2007;46(6):687-700.

44. Dugoua J, Machado M, Zhu X, Chen X, Koren G, Einarson T. Probiotic safety in pregnancy: a systematic review and meta-analysis of randomized controlled trials of Lactobacillus, Bifidobacterium, and Saccharomyces spp. J Obstet Gynaecol Can. 2009;31(6):542-52.

45. Ford A, Malfertheiner P, Giguere M, Santana J, Khan M, Moayyedi P. Adverse events with bismuth salts for Helicobacter pylori eradication: systematic review and meta-analysis. World J Gastroenterol. 2008;14(48):7361-70.

46. Gehling M, Tryba M. Risks and side-effects of intrathecal morphine combined with spinal anaesthesia: a metaanalysis. Anaesthesia. 2009;64(6):643-51.

47. Johansson K, Neovius K, DeSantis S, Rössner S, Neovius M. Discontinuation due to adverse events in randomized trials of orlistat, sibutramine and rimonabant: a meta-analysis. Obes Rev. 2009;10(5):564-75.

48. Lakhdar R, Al-Mallah M, Lanfear D. Safety and tolerability of angiotensin-converting enzyme inhibitor versus the combination of angiotensin-converting enzyme inhibitor and angiotensin receptor blocker in patients with left ventricular dysfunction: a systematic review and meta-analysis of randomized controlled trials. J Card Fail. 2008;14(3):181-8.

49. Luykx J, Mason M, Ferrari M, Carpay J. Are migraineurs at increased risk of adverse drug responses? a meta-analytic comparison of topiramate-related adverse drug reactions in epilepsy and migraine. Clin Pharmacol Ther. 2009;85(3):283-8.

50. Mauri D, Valachis A, Polyzos I, Polyzos N, Kamposioras K, Pesce L. Osteonecrosis of the jaw and use of bisphosphonates in adjuvant breast cancer treatment: a meta-analysis. Breast Cancer Res Treat. 2009;116(3):433-9.

51. Morris T, Castrejon S, Devendrá G, Gamst A. No difference in risk for thrombocytopenia during treatment of pulmonary embolism and deep venous thrombosis with either low-molecular-weight heparin or unfractionated heparin: a metaanalysis. Chest. 2007;132(4):1131-9.

52. Phillips C, Kashani A, Ko D, Francis G, Krumholz H. Adverse effects of combination angiotensin ? receptor blockers plus angiotensin-converting enzyme inhibitors for left ventricular dysfunction: a quantitative review of data from randomized clinical trials. Arch Intern Med. 2007;167(18):1930-6.

53. Ravindran V, Rachapalli S, Choy E. Safety of mediumto long-term glucocorticoid therapy in rheumatoid arthritis: a meta-analysis. Rheumatology. 2009;48(7):807-11.

54. Rodrigo G, Nannini L, Rodríguez-Roisin R. Safety of long-acting beta-agonists in stable COPD: a systematic review. Chest. 2008;133(5):1079-87.

55. Rodrigo G, Castro-Rodriguez J, Nannini L, Plaza Moral V, Schiavi E. Tiotropium and risk for fatal and nonfatal cardiovascular events in patients with chronic obstructive pulmonary disease: systematic review with meta-analysis. Respir Med. 2009;103(10):1421-9.

56. Siegel C, Marden S, Persing S, Larson R, Sands B. Risk of lymphoma associated with combination anti-tumor necrosis factor and immunomodulator therapy for the treatment of Crohn's disease: a meta-analysis. Clin Gastroenterol Hepatol. 2009;7(8):874-81.

57. Silva M, Matthews M, Jarvis C, Nolan N, Belliveau P, Malloy M, Gandhi P. Meta-analysis of drug-induced adverse events associated with intensive-dose statin therapy. Clin Ther. 2007;29(2):253-60.

58. Singh S, Loke YK, Furberg CD. Long-term risk of cardiovascular events with rosiglitazone: a meta-analysis. JAMA. 2007 Sep 12;298(10):1189-95.

59. Singh S, Loke YK, Furberg CD. Inhaled anticholinergics and risk of major adverse cardiovascular events in patients with chronic obstructive pulmonary disease: a systematic review and meta-analysis. JAMA. 2008 Sep 24;300(12): 1439-50.

60. Singh S, Amin AV, Loke YK. Long-term use of inhaled corticosteroide and the risk of pneumonia in chronic obstructive pulmonary disease: a meta-analysis. Arch Intern Med. 2009 Feb 9;169(3):219-29.