Efficacy of Quality Criteria to Identify Potentially Harmful Information: A Cross-sectional Survey of Complementary and Alternative Medicine Web Sites

Abstract

Background: Many users search the Internet for answers to health questions. Complementary and alternative medicine (CAM) is a particularly common search topic. Because many CAM therapies do not require a clinician's prescription, false or misleading CAM information may be more dangerous than information about traditional therapies. Many quality criteria have been suggested to filter out potentially harmful online health information. However, assessing the accuracy of CAM information is uniquely challenging since CAM is generally not supported by conventional literature.

Objective: The purpose of this study is to determine whether domain-independent technical quality criteria can identify potentially harmful online CAM content.

Methods: We analyzed 150 Web sites retrieved from a search for the three most popular herbs: ginseng, ginkgo and St. John's wort and their purported uses on the ten most commonly used search engines. The presence of technical quality criteria as well as potentially harmful statements (commissions) and vital information that should have been mentioned (omissions) was recorded.

Results: Thirty-eight sites (25%) contained statements that could lead to direct physical harm if acted upon. One hundred forty five sites (97%) had omitted information. We found no relationship between technical quality criteria and potentially harmful information.

Conclusions: Current technical quality criteria do not identify potentially harmful CAM information online. Consumers should be warned to use other means of validation or to trust only known sites. Quality criteria that consider the uniqueness of CAM must be developed and validated.

Introduction

Online health information can harm as well as heal. Many quality criteria have been suggested to help consumers identify misleading, inaccurate, or harmful information. Objective quality criteria that offer a limited number of options are particularly promising since they are easier to assess. For example, it is easier to assess whether an author is identified than to determine whether the author is qualified. However, even seemingly objective quality criteria have proven unreliable without specific operational definitions [1]. Further, there is little evidence that these criteria, known as "technical criteria," actually filter out undesirable health information. The few studies that have attempted to evaluate technical criteria reported conflicting results [2-4]. If harmful information can be effectively identified, this should be publicized. If, on the other hand, currently available quality criteria cannot identify potentially harmful information, then we should caution consumers and work on finding other ways of identifying problematic information online.

In this study, we analyze Web sites that display information about complementary and alternative medicine (CAM). CAM includes "diverse medical and healthcare systems, practices and products that are not presently considered to be a part of conventional medicine," such as dietary supplements, aromatherapy, chiropractic, and homeopathy [5]. Assessing accuracy and quality of CAM Web sites poses unique challenges as there is less documented research on the efficacy of CAM products, yet use is common and the potential for harm remains. There is also no gatekeeper to control and monitor access to CAM. Consumers can choose the product and dosage without having to encounter a healthcare professional. In fact, patients often fail to report CAM use to their physicians [6]. On the other hand, consumers frequently turn to the Internet to answer questions about CAM, and trust and act upon what they see online [7]. However, CAM information online has been found to be commercially driven [8], to be poorly referenced [8], and to contain illegal claims [9], and it may therefore be dangerous to consumers [10]. The combination of accessible, unproven CAM therapies and poor quality online CAM information is dangerous.

"Accuracy is a function of whether a site reflects the use of … agreed-upon benchmark[s] such as clinical practice guidelines." [11] The accuracy of CAM information, which is often not evidence-based and lacks support from the peer-reviewed biomedical literature, is not testable. However, we can assess the potential harm of displayed information, even if we cannot verify its accuracy. Further, if information regarding the safety and efficacy of a product is available, it should be displayed.

Our previous work provides preliminary evidence that breast cancer Web sites that meet more technical quality criteria are less likely to contain false statements [12]. Motivated by a desire to help consumers, we sought to determine whether current technical quality criteria can identify potentially harmful CAM information.

Materials and Methods

Selection of Web Sites

Consumers use general-purpose search engines rather than medical sites or portals to find information, and most do not go beyond the first page of search results [13]. Therefore, we chose the ten most popular search engines (Table 1) to select Web sites that consumers are likely to encounter [14]. The three most popular herbs in the United States (in terms of dollars spent) [15], ginseng, ginkgo, and St. Johns wort, and their most common uses formed the search query. The following three queries were executed in each search engine on July 15, 2003: "ginseng and cancer," "ginkgo and memory loss," and "St. John's wort and depression." All Web sites listed on the first results page, including sponsored or paid links, were analyzed.

Table 1. Search engines used to select Web sites

Search Engine

1. References Sagaram S, et al. Inter-observer agreement for quality measures applied to online health information. In: Fieschi M, Coeira E, Li YC, editors. Medinfo. Amsterdam: IOS Press; 2004. [in press]. Fallis D, Frické M. Indicators of accuracy of consumer health information on the Internet: a study of indicators relating to information for managing fever in children in the home. J Am Med Inform Assoc 2002 Jan;9(1):73-79. [PMC] Griffiths KM, Christensen H. Quality of web based information on treatment of depression: cross sectional survey. BMJ 2000 Dec 16;321(7275):1511-1515 [PMC] Martin-facklam M, Kostrzewa M, Schubert F, Gasse C, Haefeli WE. Quality markers of drug information on the Internet: an evaluation of sites about St. John's wort. Am J Med 2002 Dec 15;113(9):740-745. ; National Center for Complementary and Alternative Medicine, National Institutes of Health. What is complementary and alternative medicine (CAM)?. 2002. URL: http://nccam.nih.gov/health/whatiscam/ [accessed 2004 Jun 24] Eisenberg DM, Kessler RC, Van Rompay MI, Kaptchuk TJ, Wilkey SA, Appel S, et al. Perceptions about complementary therapies relative to conventional therapies among adults who use both: results from a national survey. Ann Intern Med 2001 Sep 4;135(5):344-351 Fox S, Rainie L. Vital decisions: how Internet users decide what information to trust when they or their loved ones are sick. Pew Internet American Life Project. 2002 May 22. URL: http://www.pewinternet.org/pdfs/PIP_Vital_Decisions_May2002.pdf Sagaram S, Walji M, Bernstam E. Evaluating the prevalence, content and readability of complementary and alternative medicine (CAM) web pages on the internet. Proc AMIA Symp 2002:672-676. Morris CA, Avorn J. Internet marketing of herbal products. JAMA 2003 Sep 17;290(11):1505-1509. Ernst E, Schmidt K. 'Alternative' cancer cures via the Internet? Br J Cancer 2002 Aug 27;87(5):479-480. Risk A, Petersen C. Health information on the internet: quality issues and international initiatives. JAMA 2002;287(20):2713-2715. Meric F, Bernstam EV, Mirza NQ, Hunt KK, Ames FC, Ross MI, et al. Breast cancer on the world wide web: cross sectional survey of quality of information and popularity of websites. BMJ 2002 Mar 9;324(7337):577-581 [PMC] Eysenbach G, Köhler C. Does the internet harm health? Database of adverse events related to the internet has been set up. BMJ 2002 Jan 26;324(7331):239. Sullivan DE. Nielsen netratings search engine ratings. URL: http://searchenginewatch.com/reports/netratings.html [accessed 2003 Feb 7] Most popular herbs and supplements in the United States. URL: http://yoga.about.com/library/weekly/aa022501a.htm [accessed 2004 Jun 24] ; Health Improvement Institute. HIIQA definitions and abbreviations. URL: http://www.hii.org/343definitions.htm [accessed 2004 Jun 24] HTTrack. URL: http://HTTrack.com/ [accessed 2004 Jun 24] Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. ; Medical Economics Company. PDR for Herbal Medicines, 2nd edition. Montvale, NJ: Medical Economics Company; 2000. URL: http://dsc.ucsf.edu/view_pdf.php?pdf_id=23 Sloan Kettering AboutHerbs website. URL: http://www.mskcc.org/aboutherbs [accessed 2003 Apr 25] Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285(20):2612-2621. Markman M. Safety issues in using complementary and alternative medicine. J Clin Oncol 2002 Sep 15;20(18 Suppl):39S-41S ; Commission of the European Communities, Brussels. eEurope 2002: Quality Criteria for Health Related Websites. J Med Internet Res 2002 Nov 29;4(3):e15 Byrt T, Bishop J, Carlin JB. Bias, prevalence and kappa. J Clin Epidemiol 1993 May;46(5):423-429. Adler SR, Fosket JR. Disclosing complementary and alternative medicine use in the medical encounter: a qualitative study in women with breast cancer. J Fam Pract 1999 Jun;48(6):453-458. Cooke A, Gray L. Evaluating the quality of internet-based information about alternative therapies: development of the BIOME guidelines. J Public Health Med 2002 Dec;24(4):261-267. Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA 1997 Apr 16;277(15):1244-1245.