Evaluation of the scope, quality, and health literacy demand of Internet-based anal cancer information

Abstract

Objectives: As there is a dearth of information about anal cancer available at cancer centers, patients often use the Internet to search for information. This is problematic, however, because the quality of information on the Internet is variable, and the health literacy demanded is higher than the average patrons' capacity. The purposes of this study were to (1) determine the most common websites with anal cancer consumer health information, (2) identify the supportive care needs that each website addresses, and (3) evaluate the websites' quality and health literacy demand. Methods: Medical Subject Headings (MeSH) entry terms for "Anus Neoplasms" were used in Google Canada to identify websites. Seven domains of supportive care needs were defined using Fitch's Supportive Care Framework for Cancer Care. Website quality was evaluated using the DISCERN tool. Health literacy demand was assessed using readability calculators, where best practice dictates a grade 6 or lower, and the Patient Education Material Assessment Tool (PEMAT) that computes a percentage score in 2 domains, understandability and actionability, with 80% being an acceptable score. Results: Eighteen unique websites were evaluated. One website met health literacy best practices and had a "good" quality rating. Most websites addressed only 1 supportive care domain (61%), were of "fair" quality (67%), had readability scores higher than grade 6 (89%), and had PEMAT scores ranging from 41%-92% for understandability and 0-70% for actionability. Conclusion: The information gaps on anal cancer websites warrant a need for more health literate anal cancer health information on the Internet.

Headnote

Objectives: As there is a dearth of information about anal cancer available at cancer centers, patients often use the Internet to search for information. This is problematic, however, because the quality of information on the Internet is variable, and the health literacy demanded is higher than the average patrons' capacity. The purposes of this study were to (1) determine the most common websites with anal cancer consumer health information, (2) identify the supportive care needs that each website addresses, and (3) evaluate the websites' quality and health literacy demand.

Methods: Medical Subject Headings (MeSH) entry terms for "Anus Neoplasms" were used in Google Canada to identify websites. Seven domains of supportive care needs were defined using Fitch's Supportive Care Framework for Cancer Care. Website quality was evaluated using the DISCERN tool. Health literacy demand was assessed using readability calculators, where best practice dictates a grade 6 or lower, and the Patient Education Material Assessment Tool (PEMAT) that computes a percentage score in 2 domains, understandability and actionability, with 80% being an acceptable score.

Results: Eighteen unique websites were evaluated. One website met health literacy best practices and had a "good" quality rating. Most websites addressed only 1 supportive care domain (61%), were of "fair" quality (67%), had readability scores higher than grade 6 (89%), and had PEMAT scores ranging from 41%-92% for understandability and 0-70% for actionability.

Conclusion: The information gaps on anal cancer websites warrant a need for more health literate anal cancer health information on the Internet.

INTRODUCTION

Anal cancer is a relatively uncommon cancer, but its global incidence is rising [1, 2]. The most recent statistics reported by the Canadian Cancer Society in 2013 state that approximately 580 Canadians are diagnosed with the disease each year, of whom 144 die as a result [3]. The American Cancer Society estimates that there will be 8,300 new cases of anal cancer in the United States in 2019, with 1,280 people dying from the disease [4]. However, due to advances in treatment, anal cancer is being transformed from a fatal disease to one that is increasingly curable [5]. Regardless of a curable or incurable cancer diagnosis, patients, families, and caregivers have significant information needs [6-8] that can be met with consumer health information and teaching from health care providers [9, 10].

Despite the efforts that consumer health librarians make to develop and maintain hospital library collections that are high quality for the diverse patient populations they serve, gaps exist in collections on rare diseases. In the instance of anal cancer, print patient information and educational resources are scarce at cancer centres in Canada. Specialty societies do not actively develop and disseminate anal cancer-specific print materials but do provide some information online [11]. As a result, when patrons seek information about anal cancer, consumer health librarians and library volunteers search the Internet to find information on an ad hoc basis.

Ad hoc searching is problematic for three primary reasons. First, although library volunteers are trained, they might not have the skills and experience needed to vet the quality of the information that they find online [12]. This is particularly concerning because the quality of online health information is highly variable, the information is often incomplete, and frequently the information is not what patients need to make well-informed decisions [13, 14]. Moreover, ad hoc searching for information can mean that patrons receive information that does not meet the quality standards of the library.

Second, ad hoc searching may unintentionally limit the scope of information that is available to patrons, because patrons who have the benefit of browsing a library collection can see what other information they might need [15]. In the case of cancer patients and their families or caregivers, these informational needs may be diverse, as supportive care needs extend far beyond pain or symptom management and counselling [16]. If patrons have to ask for information, they might not think to ask for information that falls in alternative domains. Therefore, ad hoc searching necessitates more deliberate search strategies, and patrons may not be aware of resource topics that could be beneficial to them.

Third, a significant proportion of Canadians have low health literacy [17], and individuals with low health literacy have difficulty accessing health information, appraising its quality, and applying it to their own circumstances [18]. As such, it is likely that a large proportion of patrons will not have the skills to assess the quality and applicability of health information themselves. Accessing reliable information on rare cancers, like anal cancer, is especially challenging since there is a lack of consumer health information due to the relatively small number of patients who are affected [19].

Furthermore, it is important to take a "universal precautions approach," whereby services in health systems operate at a level where the health literacy demand is not strenuous for patients, families, and caregivers, because the assumption is all service users might have low health literacy [20]. A balance must be struck between the demands and complexities of the health system and the skills and abilities of its users [21]. To mitigate health literacy demand, plain language is one patient education strategy in which resources are written and formatted for the intended audience so that information is clearly understood and actionable [22, 23].

Considering the dearth of anal cancer resources in library collections, the global increase in anal cancer incidence [1, 2], advances in treatment [5], and the necessity for high-quality health-literate consumer health information [10], the authors sought to determine the breadth and quality of anal cancer consumer health information available on the Internet. The objectives of this study were to (1) identify the most common websites with information about anal cancer, (2) determine the supportive care needs addressed by anal cancer information found online, and (3) evaluate the quality and health literacy demand of anal cancer information found online. We further sought to establish a process to evaluate web-based consumer health information that could be reproduced by medical librarians who are interested in evaluating the quality of web-based consumer health information on other topics, including other rare diseases.

METHODS

Identifying websites with information about anal cancer

Entry terms for "Anus Neoplasms" in Medical Subject Headings (MeSH), the National Library of Medicine controlled vocabulary thesaurus for indexing articles, were used to determine the differing variations to communicate "anal cancer" in a systematic and repeatable manner. This resulted in ten MeSH-based search terms to identify websites for this evaluation: anal cancer; anal cancers; cancer, anal; cancers, anal; cancer of the anus; cancers of anus; anus cancer; anus cancers; anal neoplasm; and anus neoplasm. Although "anal neoplasm" and "anus neoplasm" are terms typically used by health care practitioners, they were included to account for patients, families, or caregivers who might use these search terms if they had heard or seen the terms in health-related reports.

Each search term was entered separately in Google Canada, and the first 15 websites that met the inclusion criteria were recorded in an Excel database. Website gathering was conducted via Google as it leads the search market in North America, with a more than 90% market share [24], and has been shown to have the highest search efficiency for health information when compared to other search engines [25]. The first 15 websites were recorded in an Excel spreadsheet to reflect search engine results page (SERP) Google data. Average traffic percentages in Google reveal that 93% of searchers only look at and select a website to view from the first SERP [26, 27], whereas traffic drops by 95% for the second SERP and continues to decrease for subsequent pages [27].

Inclusion criteria were websites that provided consumer health information on anal cancer that were published in English. Exclusion criteria included websites that were blogs, news or magazine articles, videos, books, dictionary definitions, and research or academic articles, as well as websites that were not available in English, required a membership to view the content, or had no relevance to anal cancer.

Each search term had its own sheet in the Excel spreadsheet documenting the following information: (1) title of web page, (2) name of publishing organization, (3) type of publishing organization, (4) country of publishing organization, (5) uniform resource locator (URL), (6) date of search, and (7) observations or comments. Types of publishing organizations were classified based on industry: nonprofit or charity, (private or public) company or subsidiary, professional (medical) society, government agency, or higher education. We also had a sheet for "cumulative" results, in which all websites were entered and tallied based on the number of search terms that retrieved a particular website.

Determining supportive care needs addressed

Using the Supportive Care Framework for Cancer Care [16], which categorizes the needs of cancer patients into seven domains, we documented which domains were addressed by each anal cancer website in a table. The seven domains are: (1) emotional needs: coping with anger, despair, fear, and hopelessness; (2) practical needs: navigating finances, childcare, housekeeping, and legal systems; (3) spiritual needs: reflecting on the meaning of life and the concept of pain and suffering; (4) social needs: balancing family, relationships, school, and work; (5) psychological needs: dealing with issues of self-worth, body image, and coping strategies; (6) physical needs: dealing with pain, nausea, vomiting, and fatigue; and (7) informational needs: informing the patient's, family's, or caregiver's decision making and providing information that assists in skill acquisition [16]. Three independent reviewers conducted this categorization, compared findings to ensure accuracy, and reported the categorization that they agreed upon. If disagreement occurred between the reviewers, a fourth reviewer facilitated consensus.

Evaluating the quality and health literacy demand of anal cancer information

Quality assessment. To assess the quality of the consumer health information presented in each website, the validated and freely available online DISCERN tool [28] was used to encourage reproducibility of our approach among medical librarians. This tool is designed specifically to assess the quality of consumer health information content on the Internet about treatment choices, has been designed to be used by anyone (e.g., patients, families or caregivers, health professionals), and has been widely used in academic literature [29-39].

The DISCERN tool consists of 16 questions scored on a numeric rating scale ranging from No (1 point) to Yes (5 points). Questions are divided into 3 sections: (1) reliability and trustworthiness of the source of information, (2) comprehensiveness of the information regarding treatment choice, and (3) overall quality of the website. Higher scores indicate better quality: overall scores less than 27 indicate very poor quality, 27-38 indicate poor quality, 39-56 indicate fair quality, 57-62 indicate good quality, and 63 and above indicate excellent quality [28-39]. Two independent reviewers conducted this assessment, compared findings to ensure accuracy, and reported the average of their cumulative scores.

Health literacy demand assessment. Regardless of how informative and accurate a website might be, if it is not readable or well presented, it may not be accessible to the intended audience and, thus, helpful to readers. Since the DISCERN tool exclusively examines what information a website has provided, we also assessed readability, understandability, and actionability.

The readability of each website's content was assessed using readability calculators, which are validated tools that compute grade levels of text based on vocabulary complexity [40]. Each readability calculator uses a mathematical formula to compute grade levels based on the number of words in a sentence, the number of words that have more than 2 syllables, and the number of sentences in a 100-word passage. We reported the average grade level from 7 calculators-Flesch Reading Ease, Gunning Fog, Flesch-Kincaid Grade Level, Coleman-Liau Index, SMOG Index, Automated Readability Index, and Linsear Write Formula-for each web page using free, publicly available online software [41] to encourage reproducibility of our approach among medical librarians. Readability calculators vary in their lenience and subjectivity, so taking average scores from multiple calculators is considered a strong practice. Content entry followed the US Department of Health and Human Services guidelines for reliability [42]. Plain language best practices for readability were met if websites scored at a grade level of 6 or below [43, 44].

The understandability and actionability of each website's content were assessed using the validated Patient Education Materials Assessment Tool (PEMAT) [45]. Understandability is defined as key messages of text being understood by persons of any background or health literacy level, and actionability is defined as whether persons of any background or health literacy level understand and distinguish any actions that they can take based on the presented text [45]. Since only the written content was assessed and not any multimedia (e.g., videos), we used the PEMAT-Print version. This measure consists of 26 questions that evaluate text based on 4 categories to capture understandability: (1) content, (2) literacy demand, (3) graphics, and (4) layout and typography. To account for actionability, these measures consist of how the actions were described in the text and to whom they were addressed. The PEMAT allocates a percentage score for understandability and actionability separately, with higher percentages indicating greater understandability or actionability of the text. As with the Supportive Care Needs categorization, the same 3 independent reviewers used the PEMATPrint to evaluate the understandability and actionability of each website, compared findings, and reported the average of their cumulative scores. Plain language best practices were met if websites had an understandability and actionability score of 80% or greater [46].

RESULTS

Numeric summary

The search methodology resulted in 18 websites for this evaluation: American Cancer Society [47], American Society of Colon and Rectal Surgeons [48], American Society of Clinical Oncology's CancerNet [49], American Society for Radiation Oncology's Radiation Therapy (RT) Answers [50], Bowel Cancer Australia [51], Canadian Cancer Society [52], Cancer Council Victoria [53], Cancer Research UK [54], Healthline [55], Macmillan Cancer Support [56], Mayo Clinic [57], Medicine Net [58], National Cancer Institute [59], National Health Service [60], Patient.info [61], University of California, San Francisco's UCSF Health [62], University of Pennsylvania's Oncolink [63], and WebMD [64]. Most websites identified were hosted by nonprofit or charity organizations (39%) [47, 51-54, 56, 57] and organizations based in the United States (61%) [4750, 55, 57-59, 62-64] (Table 1).

Supportive care needs addressed

The American Cancer Society website [47], Medicine Net [58], and WebMD [64] appeared most frequently, being retrieved by all ten search terms; however, of these websites, only the American Cancer Society also had subject matter that addressed all seven supportive care domains (Table 2). The website that appeared least frequently was the American Society for Radiation Oncology's RT Answers [50], which only appeared when the search term "anus cancers" was used, and it only addressed the informational supportive care domain.

Most websites solely addressed the "informational" supportive care domain [47-64]. Whereas websites from all types of organizations addressed the informational supportive care domain, only nonprofit or charity, professional society, and government agency websites addressed the social, emotional, physical, practical, and psychological supportive care domains. Less than half of the websites addressed spiritual needs (22%) [47, 52, 56, 59], psychological needs (33%) [47, 49, 52, 53, 56, 59], practical needs (44%) [47, 49, 52-54, 56, 57, 59], physical needs (44%) [47, 49, 50, 52-54, 56, 59], or emotional needs (44%) [47, 49, 50, 52-54, 56, 59]. Only 4 websites (22%) encompassed content that addressed all supportive care domains: American Cancer Society [47], Macmillan Cancer Support [56], Canadian Cancer Society [52], and National Cancer Institute [59].

Quality and health literacy demand

Overall, most websites were rated as being of "fair" quality (67%), with only the American Cancer Society website [47] receiving an "excellent" quality score of 66.5 (83%; SD: ±6.5; Table 3). Professional society websites had the highest average quality scores (50.8; 64%; SD: ±5.7) [48-50], followed by nonprofit or charity (46.9; 59%; SD: ±11.4) [47, 51-54, 56, 57]; government (45.8; 57%; SD: ±2.8) [59, 60]; higher education (41.0; 51%; SD: ±3.0) [62, 63]; and company or subsidiary (35.8; 45%; SD: ±8.6) [55, 58, 61, 64] websites. The website that received the lowest quality score was Healthline [55], which received a "very poor" quality rating (25.5; 32%; SD: ±2.5; Table 3).

DISCERN items for which anal cancer websites fell short were: describing if no treatment is used (89% received a "very poor" score [49-64]); referencing areas of uncertainty (56% "very poor" [50, 52-57, 59, 62]; 22% "poor" [51, 52, 60, 63]); describing benefits of treatment (50% "very poor" [51, 54-59, 62, 64]; 39% "poor" [49, 50, 52, 53, 60, 61, 63]); describing how treatment choices affect quality of life (50% "very poor" [48, 51, 55, 58, 59, 61, 62-64]; 17% "poor" [50, 52, 60]); and providing support for shared decision making (44% "very poor" [50, 51, 55, 57, 58, 61, 62, 64]; 28% "poor" [48, 53, 54, 60, 63]) (Figure 1 in the supplemental appendix).

The health literacy demand of these websites was high, as only one website, Cancer Research UK [54], met plain language best practices. Cancer Research UK [54] had content written at a grade level of 5 and an understandability score of 87.7% (Table 3). Although 2 other websites also surpassed the best practices threshold of an 80% understandability score (American Society of Clinical Oncology [49] and National Cancer Institute [59]), their readability scores were high, at a grade level of 8. Readability scores for all websites ranged from a grade level of 5 to 13, with most websites having content written above a grade level of 6 (89%) [49-53, 55, 57-64].

The PEMAT understandability scores for all websites ranged from 41%-92%. The National Cancer Institute had the highest score (92%; SD: ±3%) [59], and Patient.info had the lowest score (41%; SD: ±4%) [61]. The PEMAT actionability scores for all websites ranged from 0-70%. The Mayo Clinic website had the highest actionability score (70%; SD: ±8%) [57], and Patient.info had the lowest score (0; SD: ±0) [61]. Overall, websites by companies or subsidiaries imposed the greatest health literacy demand on patients [55, 58, 61, 64]. On average, the readability scores of these websites were associated with a grade level of 11 (SD: ±1.5), the average understandability score was 49.8% (SD: ±6.3%), and the average actionability score was 11.7% (SD: ±9.9%).

DISCUSSION

This evaluation demonstrates that most consumer health information websites about anal cancer are of low quality and require readers to have high health literacy. This is consistent with previous evaluations of online patient information for a diverse range of other medical conditions [31-38]. Only the Cancer Research UK website met plain language best practices and was rated "good" quality [54]. However, this website did not address all supportive care needs. Moreover, most websites did not address a breadth of supportive care needs, particularly spiritual or psychological needs. Most websites whose scope included all supportive care domains imposed a moderate health literacy demand and were of "fair" quality [52, 56, 59], except for one website that had an "excellent" quality score [47]. Websites by private and public companies or subsidiaries-such as Healthline, Patient.info, and WebMD-had the poorest quality, readability, understandability, and actionability [55, 58, 61, 64]. Therefore, our findings suggest that online anal cancer information is not accessible for most patients and families or caregivers, considering the prevalence of low health literacy.

When online information is accessible, patients report more informed decision making concerning health or treatment issues and greater willingness to make health behavior changes to improve health status [65]. The results of the present study indicate that understandable and actionable anal cancer resources are needed. This is likely the case for other rare cancers and conditions, which have fewer resources available due, in large part, to historically limited attention and investment [66].

Implications for medical librarians include: (1) collaborating with subject matter experts to develop understandable and actionable patient information about anal cancer; (2) advocating for evaluations of quality and health literacy demand before publishing any patient, family, and caregiver information online; and (3) utilizing this study to serve as a framework when developing Internet "pathfinder" documents for rare cancers, such as anal cancer, or other conditions for which there are gaps in library collections and internal resources are scarce [67]. In the case of anal cancer, these results can serve as guidance in place of ad hoc searching by providing a list of websites to recommend to patrons that are high quality, impose low health literacy demand, and address a range of supportive care needs.

A limitation of this study was that it utilized a convenience sample of websites (top 15 in the search yield) rather than sampling all websites. The drawbacks of convenience sampling are well documented and include lack of generalizability and insufficient power [68].

This study reveals several informational gaps in anal cancer websites and highlights a need for more understandable and actionable information. Most websites solely addressed the "informational" supportive care domain, with only four websites addressing all domains. Most websites had high health literacy demand, as they did not meet plain language best practices. The only website (Cancer Research UK) that met plain language best practices had a "good" quality rating and addressed five of the seven supportive care domains. This further reveals a need for comprehensive patient resources, particularly information pertaining to the spiritual and psychological needs of patients with anal cancer. Finally, this study provides a useful framework for identifying and evaluating the quality of consumer health information on the Internet and can be adapted to assess the quality and scope of informational websites for other cancer types.

COMPETING INTERESTS STATEMENT

None declared.

ACKNOWLEDGMENTS

This work was supported by the Princess Margaret Cancer Foundation.

Sidebar

SUPPLEMENTAL FILE

* Appendix: Figure 1 Average of each DISCERN item for website quality

AUTHORS' AFFILIATIONS

Rebecca Charow, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Michelle Snow, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Sameera Fathima, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Meredith E. Giuliani, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Kate McEwan, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Jordana Winegust, [email protected], Princess Margaret Cancer Centre, University Health Network, Toronto, ON, Canada

Janet Papadakos, [email protected], Princess Margaret Cancer Centre, University Health Network, and Patient Education, Cancer Care Ontario, Toronto, ON, Canada

Received October 2017; accepted June 2019

References

REFERENCES

1. Islami F, Ferlay J, Lortet-Tieulent J, Bray F, Jemal A. International trends in anal cancer incidence rates. Int J Epidemiol. 2017 Jun;46(3):924-38. DOI: http://dx.doi.org/10.1093/iie/dyw276.

2. Pricolo VE, Bonvini M, Abelli CF. Patterns of care for anal cancer in the United States - a comparison between academic and community cancer centers. BMC Cancer. 2018 May 16;18(1):567. DOI: http://dx.doi.org/10.1186/s12885-018-4488-1.

3. Canadian Cancer Society. Anal cancer statistics [Internet]. Toronto, ON, Canada: Canadian Cancer Society; 2019 [cited 12 Feb 2019]. <http: / /www.cancer.ca/en/cancerinformation/ cancer-type / anal / statistics / >.

4. American Cancer Society. Key statistics for anal cancer [Internet]. Atlanta, GA: The Society; 2019 [cited 12 Feb 2019]. <https:// www.cancer.org/cancer/anal-cancer/about/what-is-key-statistics.html>.

5. Denlinger CS, Barsevick AM. The challenges of colorectal cancer survivorship. J Natl Compr Canc Netw. 2009 Sep;7(8):883-93; quiz 894.

6. Hesse BW, Arora NK, Burke Beckjord E, Finney Rutten LJ. Information support for cancer survivors. Cancer. 2008 Jun 1;112(11 suppl):2529-40. DOI: http:/ / dx.doi.org/10.1002/cncr.23445.

7. Mistry A, Wilson S, Priestman T, Damery S, Haque M. How do the information needs of cancer patients differ at different stages of the cancer journey? a cross-sectional survey. JRSM Short Rep. 2010 Sep 15;1(4):30. DOI: http:// dx.doi.org/10.1258/shorts.2010.010032.

8. Papadakos J, Bussiere-Côté S, Abdelmutti N, Catton P, Friedman AJ, Massey C, Urowitz S, Ferguson SE. Informational needs of gynecologic cancer survivors. Gynecol Oncol. 2012 Mar;124(3):452-7. DOI: http:// dx.doi.org/10.1016/i.vgyno.2011.10.030.

9. Friedman AJ, Cosby R, Boyko S, Hatton-Bauer J, Turnbull G. Effective teaching strategies and methods of delivery for patient education: a systematic review and practice guideline recommendations. J Cancer Educ. 2011 Mar;26(1):12-21. DOI: http://dx.doi.org/10.1007/s13187-010-0183-x.

10. O'Dell R. Appraisal skills, health literacy and the patient-provider relationship: considerations as the health care consumer turns to the Internet to inform their care. Online J Public Health Inform. 2012;4(1):pii: ojphi.v4i1.3684. DOI: http:/ / dx.doi.org/10.5210/ojphi.v4i1.3684.

11. Colorectal Cancer Canada. CCAC documentation: colorectal cancer & you [Internet]. 4th ed. Toronto, ON, Canada: Colorectal Cancer Canada; 2018 [cited 12 Feb 2019]. <http: / / archive.colorectalcancercanada.com/ en/advocacy/c cac-materials/ >.

12. McDiarmid M, Auster E. Using volunteers in Ontario hospital libraries: views of library managers. J Med Libr Assoc. 2005 Apr;93(2):253-62.

13. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the World Wide Web: a systematic review. JAMA. 2002 May 22-29;287(20):2691-700.

14. Wasserman M, Baxter N, Rosen B, Burnstein M, Halverson AL. Systematic review of Internet patient information on colorectal cancer surgery. Dis Colon Rectum. 2014 Jan;57(1):64-9. DOI: http://dx.doi.org/10.1097/DCR.0000000000000011.

15. Papadakos J, Trang A, Cyr AB, Abdelmutti N, Giuliani ME, Snow M, McCurdie T, Pulandiran M, Urowitz S, Wiljer D. Deconstructing cancer patient information seeking in a consumer health library toward developing a virtual information consult for cancer patients and their caregivers: a qualitative, instrumental case study. JMIR Cancer. 2017 May 24;3(1):e6. DOI: http:// dx.doi.org/10.2196/cancer.6933.

16. Fitch M. Supportive care framework. Can Oncol Nurs J. 2008 Winter;18(1):6-24.

17. Rootman I, Gordon-El-Bihbety D. A vision for a health literate Canada: report of the Expert Panel on Health Literacy [Internet]. Ottawa, ON, Canada: Canadian Public Health Association; 2008 [cited 12 Feb 2019]. <https:/ /www.cpha.ca/vision-health-literate-canada-report-expert-panel-health-literacy>.

18. Diviani N, van den Putte B, Giani S, van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res. 2015 May 7;17(5):e112. DOI: http:/ / dx.doi.org/10.2196/imir.4018.

19. Partarrieu J. A new approach to rare cancers [Internet]. London, UK: European Oncology Nursing Society; 2015 [cited 8 Jan 2018]. <https:/ /www.cancernurse.eu/documents/magazine/201 5SpringSummer/EONSMagazine2015SpringSummerPage2 1.pdf>.

20. DeWalt DA CL, Hawk VH, Broucksou KA, Hink A, Rudd R, Brach C. Health literacy universal precautions toolkit [Internet]. Rockville, MD: Agency for Healthcare Research and Quality; 2010 [cited 12 Feb 2019]. <https: / / www. ahrq .gov/ sites / default / files / wysiwyg / pr ofessionals/ quality-patient-safety / quality-resources/tools/literacy-toolkit/healtbliteracytoolkit.pdf>.

21. Parker R. 6, Measuring health literacy: what? so what? now what? In: Institute of Medicine, ed. Measures of health literacy: workshop summary. Washington, DC: National Academies Press (US); 2009.

22. Coleman CA, Hudson S, Maine LL. Health literacy practices and educational competencies for health professionals: a consensus study. J Health Commun. 2013;18(suppl 1):82-102. DOI: http://dx.doi.org/10.1080/10810730.2013.829538.

23. Wizowski L, Harper T, Hutchings T. Writing health information for patients and families: a guide to developing educational materials that promote health literacy. 4th ed. Hamilton, ON, Canada: Hamilton Health Sciences; 2014.

24. Desjardins J. How Google retains more than 90% of market share [Internet]. New York, NY: Business Insider; 2018 [cited 12 Feb 2019]. <https:/ /www.businessinsider.com/how-google-retains-more-than-90-of-market-share-2018-4>.

25. Neuman HB, Cabral C, Charlson ME, Temple LK. Is Internet information adequate to facilitate surgical decision-making in familial adenomatous polyposis? Dis Colon Rectum. 2007 Dec;50(12):2135-41.

26. Allan A. Research methods for advanced web searching. In: Greenfield T, Greener S, eds. Research methods for postgraduates. 3rd ed. Chichester, UK; Hoboken, NJ: John Wiley & Sons; 2016. p. 94-112.

27. Chitika. The value of Google result positioning. Westborough, MA: Chitika Online Advertising Network; 2013.

28. Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53(2):105-11.

29. Bruce JG, Tucholka JL, Steffens NM, Neuman HB. Quality of online information to support patient decision-making in breast cancer surgery. J Surg Oncol. 2015 Nov;112(6):575-80. DOI: http://dx.doi.org/10.1002/iso.24046.

30. Carlsson T, Axelsson O. Patient information websites about medically induced second-trimester abortions: a descriptive study of quality, suitability, and issues. J Med Internet Res. 2017 Jan;19(1):e8. DOI: http:// dx.doi.org/10.2196/imir.6380.

31. Daraz L, MacDermid JC, Wilkins S, Gibson J, Shaw L. The quality of websites addressing fibromyalgia: an assessment of quality and readability using standardised tools. BMJ Open. 2011 Jul 31;1(1):e000152. DOI: http://dx.doi.org/10.1136/bmiopen-2011-000152.

32. Doruk C, Enver N, Çaytemel B, Azezli E, Başaran B. Readibility, understandability, and quality of online education materials for vocal fold nodules. J Voice. 2018 Sep 18. pii: S0892-1997(18):30319-9. DOI: http:/ / dx.doi.org/10.1016/iivoice.2018.08.015.

33. Kirby PL, Reynolds KA, Walker JR, Furer P, Pryor TAM. Evaluating the quality of perinatal anxiety information available online. 2018 Dec;21(6):813-20. DOI: http://dx.doi.org/10.1007/s00737-018-0875-5.

34. Lam CG, Roter DL, Cohen KJ. Survey of quality, readability and social research of websites on osteosarcoma in adolescents. Patient Educ Couns. 2013 Jan;90(1):82-7. DOI: http://dx.doi.org/10.1016/i.pec.2012.08.006.

35. Leira Y, Castelo-Baz P, Pérez-Sayáns M, Blanco J, Lorenzo-Pouso AI. Available patient-centered Internet information on peri-implantitis. can our patients understand it? Clin Oral Investig. 2 019 Apr;23(4):1569-74. DOI: http://dx.doi.org/10.1007/s00784-018-2583-y.

36. Prasanth AS, Jayaraiah U, Mohanappirian R, Seneviratne SA. Assessment of the quality of patient oriented information over Internet on testicular cancer. BMC Cancer. 2018 May 2;18(1):491. DOI: http://dx.doi.org/10.1186/s12885-018-4436-0.

37. San Giorgi MRM, de Groot OSD, Dikkers FG. Quality and readability assessment of websites related to recurrent respiratory papillomatosis. Laryngoscope. 2017 Oct;127(10):2293-7. DOI: http:// dx.doi.org/10.1002/lary.26521.

38. Seymour N, Lakhani R, Hartley B, Cochrane L, Jephson C. Cochlear implantation: an assessment of quality and readability of web-based information aimed at patients. Cochlear Implants Int. 2015;16(6):321-5. DOI: http://dx.doi.org/10.1179/1754762815Y.0000000015.

39. Som R, Gunawardana NP. Internet chemotherapy information is of good quality: assessment with the DISCERN tool. Br J Cancer. 2012 Jul 10;107(2):403; author reply, 404. DOI: http://dx.doi.org/10.1038/bic.2012.223.

40. Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006 Jun;33(3):352-73.

41. Eltorai AE, Naqvi SS, Ghanian S, Eberson CP, Weiss AP, Born CT, Daniels AH. Readability of invasive procedure consent forms. Clin Transl Sci. 2015 Dec;8(6):830-3. DOI: http://dx.doi.org/10.1111/cts.12364.

42. McGee J, McGee & Evers Consulting. Part 7: using readability formulas: a cautionary note. In: Toolkit for making written material clear and effective [Internet]. Washington, DC: US Department of Health and Human Services; 2012 [cited 12 Feb 2019]. <https://www.cms.gov/outreach-and-education/outreach/writtenmaterialstoolkit/downloads/t oolkitpart07.pdf>.

43. The Joint Commission. Advancing effective communication, cultural competence, and patient- and family-centered care: a roadmap for hospitals [Internet]. Oakbrook Terrace, IL: The Commission; 2010 [cited 12 Feb 2019]. <https: / /www.iointcommission.org/assets/1 /6/ARoadm apforHospitalsfinalversion727.pdf>.

44. Weiss B. Health literacy: a manual for clinicians. Chicago, IL: American Medical Association; 2008.

45. Shoemaker S, Wolf MS, Brach C. The Patient Education Materials Assessment Tool (PEMAT) and user's guide: an instrument to assess the understandability and actionability of print and audiovisual patient education materials [Internet]. Version 1.0. Rockville, MD: Agency for Healthcare Research and Quality; 2014 [cited 12 Feb 2019]. <https: / / www. ahrq .gov/ sites / default / files / publications/ files/pemat guide.pdf>.

46. Papadakos JK, Charow RC, Papadakos C, Moody LJ, Giuliani ME. Evaluating cancer patient-reported outcome measures: readability and implications for clinical use. Cancer 2019 Apr 15;125(8):1350-6. DOI: http:// dx.doi.org/10.1002/cncr.31928.

47. American Cancer Society. What is anal cancer? [Internet]. Atlanta, GA: The Society; 2019 [cited 12 Feb 2019]. <https://www.cancer.org/cancer/anal-cancer/about /what-is-anal-cancer.html>.

48. American Society of Colon and Rectal Surgeons. Anal cancer [Internet]. Oakbrook Terrace, IL: The Society; 2019 [cited 12 Feb 2019]. <https: / / www.fascrs.org / patients/disease-condition/anal-cancer>.

49. American Society of Clinical Oncology. CancerNet: anal cancer guide [Internet]. Alexandria, VA: The Society; 2018 [cited 12 Feb 2019]. <https: / / www.cancer.net/ cancer-types/ anal-cancer / view-all>.

50. American Society for Radiation Oncology. RT Answers: colon, rectum and anus cancers [Internet]. Arlington, VA: The Society [cited 11 Jul 2019]. <http: / / www.rtanswers.org / Cancer-Types/Colon,-Rectum-and-Anus-Cancers>.

51. Bowel Cancer Australia. What is anal cancer? [Internet]. North Sydney, NSW, Australia: Bowel Cancer Australia; 2017 [cited 12 Feb 2019]. <https:/ /www.bowelcanceraustralia.org/what-is-anal-cancer>.

52. Canadian Cancer Society. What is anal cancer? [Internet]. Toronto, ON: The Society; 2019 [cited 12 Feb 2019]. <http: / / www.cancer.ca / en / cancer-information / cancer-type / anal / anal-cancer / >.

53. Cancer Council Victoria. Anal cancer overview [Internet]. Melbourne, VIC, The Council; 2019 [cited 12 Feb 2019]. <https:/ /www.cancervic.org.au/cancer-information/ types-of-cancer / anal cancer / anal-cancer-overview.html>.

54. Cancer Research UK. Anal cancer [Internet]. London, UK: Cancer Research UK; 2016 [cited 12 Feb 2019]. <https://www.cancerresearchuk.org/about-cancer/anal-cancer>.

55. Healthline. Anal cancer [Internet]. San Francisco, CA: Healthline Media; 2018 [cited 12 Feb 2019]. <http://www.healthline.com/health/anal-cancer>.

56. Macmillan Cancer Support. Anal cancer [Internet]. London, UK: Macmillan Cancer Support; 2019 [cited 12 Feb 2019]. <https: / / www.macmillan.org.uk / information-and-support/anal-cancer>.

57. Mayo Clinic. Anal cancer [Internet]. Rochester, MN: Mayo Foundation for Medical Education and Research; 2019 [cited 12 Feb 2019]. <https:/ /www.mayoclinic.org/diseases-conditions/anal-cancer / symptoms-causes/ syc-20354140>.

58. Zatzkin JB, Davis CP. Anal cancer [Internet]. New York, NY: MedicineNet; 2018 [cited 12 Feb 2019]. <https: / / www.medicinenet.com/ anal cancer / article.htm>.

59. National Cancer Institute. Anal cancer treatment (PDQ) - patient version [Internet]. Bethesda, MD: The Institute; 2018 [cited 12 Feb 2019]. <https: / / www. cancer .gov/ types / anal / patient / anal-treatment-pdq>.

60. National Health Service. Anal cancer [Internet]. London, UK: The Service; 2018 [cited 12 Feb 2019]. <https: / /www.nhs.uk/conditions / anal-cancer/>.

61. Tidy C. Anal carcinoma [Internet]. London, UK: Patient.info; 2017 [cited 12 Feb 2019]. <https://patient.info/doctor/anal-carcinoma>.

62. UCSF Health. Anal cancer [Internet]. San Francisco, CA: Regents of the University of California; 2019 [cited 12 Feb 2019]. <https: / /www.ucsfhealth.org/conditions/anal cancer/>.

63. OncoLink Penn Medicine. All about anal cancer [Internet]. Philadelphia, PA: University of Pennsylvania; 2019 [cited 12 Feb 2019]. <https: / / www.oncolink.org / cancers/gastrointestinal/ anal -cancer / all-about-anal-cancer>.

64. WebMD. What is anal cancer? [Internet]. New York, NY: WebMD; 2017 [cited 12 Feb 2019]. <https: / / www.webmd.com/ cancer / what-is-anal-cancer#1>.

65. Wang J, Ashvetiya T, Quaye E, Parakh K, Martin SS. Online health searches and their perceived effects on patients and patient-clinician relationships: a systematic review. Am J Med. 2018 Oct;131(10):1250.e1-1250.e10. DOI: http:// dx.doi.org/10.1016/i.amimed.2018.04.019.

66. American Cancer Society. Cancer facts & figures 2017 [Internet]. Atlanta, GA: The Society; 2017 [cited 12 Feb 2019]. <https:// www.cancer.org/content/dam/ cancer-org/ research / cancer-facts-and-statistics/ annual-cancer-facts-and-figures/2017/cancer-facts-and-figures-2017.pdf>.

67. Williams MD, Gish KW, Giuse NB, Sathe NA, Carrell DL. The Patient Informatics Consult Service (PICS): an approach for a patient-centered service. Bull Med Libr Assoc. 2001 Apr;89(2):185-93.

68. Bornstein MH, Jager J, Putnick DL. Sampling in developmental science: situations, shortcomings, solutions, and standards. Dev Rev. 2013 Dec;33(4):357-70.