Into the gray: a modified approach to citation analysis to better understand research impact

Abstract

[...]we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation. [...]by moving beyond basic citation counts and coding how the paper was cited, we were able to provide a richer understanding of its impact on subsequent research, as well as policy and practice guidance regarding screening for intimate partner violence.

DOI: http://dx.doi.Org/10.3163/1536-5050 .103.1.010

Introduction

Academic authors and funders often want to know the "impact" of their publications, and this impact is generally judged by how and where the paper is cited in other academic works. This limited appraisal has been expanded in recent years as many are beginning to argue that nonacademic publishing venues should be included in assessing the impact of academic publications. This is an issue of particular concern with the growing emphasis on "knowledge translation'' from the scientific literature to policy and practice applications [1-3] and to sources other than the traditional peer-reviewed and indexed venues, in other words, translation into the "gray literature'' [4].

In this comment and opinion piece, the authors describe the process of developing and applying a "modified citation analysis'' that builds on existing methods of examining a research paper's impact in two key ways: (1) by deliberately including gray literature in the citation analysis search process, and (2) by including quantitative and qualitative methods of analysis to gain a better understanding of how a research paper was used. By broadening the search and deepening the level of analysis, we suggest this new approach can better assess the impact of a given research paper-both within and outside of traditional peer-reviewed venues. We begin with a review of gray literature and then describe current methods for analyzing the impact of a research paper. Finally, we use a specific example to describe our new approach, highlight its potential for evolving the field of citation and impact analysis, and discuss future refinements and evaluation.

Gray literature

Gray literature is an important aspect of scientific evidence as it "produc [es] and distribut[es] the seeds of new knowledge'' [5]. Current definitions of gray literature are overly broad and lack clear lines of distinction between different producers, users, and dissemination techniques (Table 1). A commonly cited definition is: "Information produced on all levels of government, academia, business and industry in electronic and print formats not controlled by commercial publishing; i.e., where publishing is not the primary activity of the producing body'' [6]. Another definition proposed by the International Journal on Grey Literature is: "the information and resources that do not categorically fall into what is available via standard, traditional or commercial publishing channels'' [7]. Even these well-respected definitions provide only a vague understanding of what is classified as gray literature.

The distribution of gray literature in a multitude of mediums by individual scholars and scientists, research institutes, and community and nonprofit organizations has become widespread since the advent of online repositories and other open access venues [8-10]. Organizations produce policy briefs, issue briefs, or technical reports on specific content areas (e.g., health, environment, and poverty) meant to inform decision making that may not enter formal publication venues. This can be attributed to a bias toward publishing larger studies with specific results (i.e., positive, novel, and generalizable), the considerable time lag between research production and publication [11], or the production of gray literature for internal organization purposes.

Gray literature as a key dissemination strategy. Gray literature remains underused and undervalued. However, its importance and relevance is becoming apparent in areas such as policy development and research [12-14]. In the medical field, the well-regarded Cochrane systematic evidence review process requires the inclusion of any relevant gray literature as part of the evidence base [15].

The exclusion of gray literature can skew the results of research syntheses that may have important ramifications for the accumulation and dissemination of scientific knowledge. For example, McAuley et al. [16] and Hopewell et al. [4] argue that the exclusion of gray literature can influence the results of meta-analyses. In one example, by extending a search to include literature outside "mainstream sources,'' an estimated 29.2% more sources were found [17]. Gray literature is also important in many fields of research in both understanding the theoretical underpinnings of findings [18] and identifying research gaps that produce research questions [19]. This is especially true in health research, where findings published in the gray literature often jhforrn the development of clinical trials [16].

Gray literature is extremely valuable in providing timely communication on complex issues-often using simple, actionable, and comprehensive language-making it a key resource for stakeholders outside of academia [20, 21]. These works often include important information on context, policy decisions, and public interest that are of particular value to decision makers [22]. It is apparent that while the amount and influence of gray literature is dramatically increasing, quality assurance, search methods, and impact measures for gray literature have not kept pace.

Quality of gray literature. The sheer volume and accessibility of gray literature also raises the issue of the quality of these resources [23]. With the general absence of peer review, the only way to ensure the quality of gray literature is through critical assessment. Although the producers of gray literature may be experienced and knowledgeable in their field, there is essentially no formal quality control. There have been suggestions to improve the quality of the more "traditional" types of gray literature such as technical and evaluation reports, strategic plans, policy briefs, and others. An example is to include standard information in documents such as: (1) the process of creating the product, (2) the review process (if any), and (3) any conflicts of interest [24]. The Grey Literature International Steering Committee provides guidelines on what scientific and technical reports should look like. They should include a peer-review process, ethical considerations, and publishing and editorial recommendations [25]. However, given the unregulated space that gray literature inhabits, it is unrealistic to expect all who create it to conform to such standards, leaving the end user responsible for assessing its quality.

Access to gray literature. Searching for gray literature is challenging and can be time consuming. There is currently no accepted strategy or protocol for doing so. While some gray literature becomes incorporated in the published literature and vice versa, much gray literature remains "fugitive" [26] and difficult to capture. Since gray literature falls under the broad definition of "materials not published in a traditional way,'' it can be found in numerous places and requires many different search strategies to collect it. Although some of the commercially available databases, such as Web of Science and Scopus, contain gray literature (e.g., dissertations, conference proceedings), this is not their primary function, and they likely capture only a fraction of existing material. Google Scholar is a good adjunct to these but similarly does not have a primary focus on gray literature. As its search algorithms are unknown, it is difficult to understand and assess what has been included and excluded from searching Google Scholar.

Another challenge with gray literature is the lack of bibliographic control or classification [27]. Unlike scholarly works such as journal articles, most gray literature is not indexed in databases. Although some databases do include gray literature, poor or inconsistent choice of titles or keywords can make such documents difficult to source, catalog, and store [28]. The "scattered" and disorganized nature of gray literature presents particular difficulty for those wishing to incorporate it into a systematic and comprehensive search strategy, as is required for many types of literature review.

Current methods for analyzing citation impact

Several methods of analyzing individual study citations are currently used to determine impact in terms of penetration into the broader academic literature. These include methods such as citation analysis, co-citation analysis, bibliographic coupling, and a co-citation-bibliographic coupling hybrid. In general, the purpose of these types of analysis is to track citations of publications and, in some methods, group similar papers together. Although these methods of impact analysis are widely used, they fail to capture gray literature, and all are based solely on counts or quantitative comparisons of citations. That is, none delve deeper to explore liozv (as opposed to simply liozv often) a research paper has been used.

A relatively new method of citation analysis is the Becker Medical Library Model for Assessment of Research (Becker Model), which analyzes research impact through indicators such as "contribution to the knowledgebase'' and "change in practice'' by tracking diffusion of research output and activities [29]. The Becker Model suggests locating gray literature materials such as non- peerreviewed journal articles, trade publications, and supplemental materials but does not assist in locating these items, and it still relies solely on quantitative counts of research outputs, albeit of more types. (For more information on this approach, visit https://becker .wustl.edu / impact-assessment.)

Modified citation analysis. We were interested in tracking the impact of a specific research article published in a high-impact, peer-reviewed journal [30]. This was the first randomized controlled trial (RCT) of universal screening in health care settings for exposure to intimate partner violence (IPV). We sought to examine how this new evidence was taken up in scholarly literature, as well as in broader policy, practice, and advocacy. (For a description of the findings of that analysis, see Wathen et al. [31].)

After reviewing current methods for citation analysis (as above), we determined that none of these could effectively track the influence of a single paper through both commercially published and gray literature. For this reason, we developed a "modified citation analysis'' method that would capture both traditional and gray literature and allow both quantitative and qualitative interpretive analysis on extracted data. Such an approach was important because (1) we knew that the content of the paper was of interest and contentious in the academic realm and beyond; therefore, limiting our search to only commercially published sources would not provide an accurate portrayal of how the findings were used, and (2) we were interested in how the paper was interpreted and used; therefore, citation counts alone would be insufficient.

We searched the scholarly and academic and gray literature for sources with high potential to influence policy, practice, or further research. Since we were primarily interested in the various uses of the study findings, we did not assess source quality or perform any formal results synthesis of the included materials. Preliminary scans of the literature helped determine the most effective subject headings and keywords or phrases to use for these searches. These terms were then applied using the appropriate database or other online search tools to retrieve articles, documents, media reports, or other items that specifically cited or discussed the MacMillan et al. paper [30]. All retrieved results that met inclusion criteria were stored in a reference management program (Figure 1).

The gray literature was searched using multiple resources, such as multidisciplinary databases ( Scopus), websites (MedlinePlus), and point-of-care tools (MDConsult, UpToDate). Major health care associations and professional organizations likely to include related content were identified, and their websites were individually searched. News media reports were searched by using news databases, including Factiva, Lexis Nexis, Google News, and Proquest Canadian Major Dailies.

"Traditional" citation analysis statistics were computed using SPSS 20.0. These included the number of sources citing the MacMillan et al. paper [30] (by search method) and the number of MacMillan et al. [30] citations per source. Data analysis took place in three steps according to the specific research questions guiding the analysis: (1) analyzing content of text extractions, (2) categorizing sources regarding their position on IPV screening, and (3) coding sources regarding how they defined IPV screening. Table 2 illustrates our coding process.

Adapting this approach. This method is theoretically adaptable to other contexts where tracing the uptake and interpretation of new knowledge would benefit from more than quantitative citation counts and inter-linkages. The gray literature search strategy would need to be specifically tailored to the area of interest and could involve more extensive searching depending on the discipline and topic area. For this reason, we recommend that experts in the field be involved in the process and that the searches be undertaken by trained information professionals.

In addition to a tailored search strategy, the quantitative and qualitative coding of the text citing the original document will also depend on the specific questions being addressed. Some codes may apply to any paper; for example, anyone might code whether the paper is cited for a methodological or theoretical point, or might examine whether the citation appears in the "Introduction," "Method," or "Discussion" sections. Most codes, however, will be highly specific to the goals of the analysis.

Discussion

Our modified citation analysis has several advantages over more mainstream methods of citation analysis. By expanding our search strategy, we were able to find more citing sources. Specifically, using search techniques designed to track citations in the commercially published academic literature (e.g., Google Scholar's "cited by'' tool and Web of Science), we found eighty sources citing the MacMillan et al. paper [30]. The gray literature searches (e.g., Google, news databases, and association websites) found an additional twenty-nine sources. Second, by moving beyond basic citation counts and coding how the paper was cited, we were able to provide a richer understanding of its impact on subsequent research, as well as policy and practice guidance regarding screening for intimate partner violence. It also provided a way to assess the evolution of the main debates in this area (for example, confusion caused by varying definitions of "screening").

This method requires additional resources. The much broader range of potential search venues demands more time and expertise. Delving into gray literature is a challenging task and requires planning and coordination, including consideration-specific inclusion /

exclusion searching. Unlike database searching, common nomenclatures rarely exist for searching diverse gray literature sources; therefore, the concept of consistency in search terms across sources is difficult to achieve. It is important to document all search strategies used in each source to attempt reproducibility in the ever- changing nature of online resources. Similarly, the addition of a qualitative analysis component necessitates multiple researchers to undertake coding, analysis, and interpretation that is rigorous and adheres to qualitative methodology standards [32]. More research using this method is required to assess its validity and utility across different applications.

Conclusion

The modified citation analysis approach appears well suited to report how new research evidence is taken up, interpreted, and used in published material, a quality that is valuable to those attempting to assess the effectiveness of knowledge translation strategies [3, 33]. We believe it is a useful extension of traditional citation analysis methods and addresses some of the gaps in the proposed Becker Model [29]. Since there are no validated methods for conducting this method of citation identification, we created a new search and analysis strategy, building on the growing importance and acknowledgment of searching gray literature [34, 35].

This new approach adds to the ongoing discussion regarding the inclusion and analysis of gray literature resources in searching for different types of "evidence" to include in both academic research, practice, and policy contexts. We hope that it encourages a more holistic approach to assessing "impact" of new research evidence.

References

References

1. Davis D, Evans M, Jadad A, Perrier L, Rath D, Ryan D, Sibbald G, Straus S, Rappolt S, Wowk M, Zwarentstein M. The case for knowledge translation: shortening the journey from evidence to effect. BMJ. 2003 Jul 5;327(7405):33-5.

2. Grol R, Grimshaw J. From best evidence to best practice: effective implementation of change in patients' care. Lancet. 2003 Oct 11;362(9391): 1225-30.

3. Kothari A, Wathen CN. A critical second look at integrated knowledge translation. Health Policy. 2013 Feb; 109:189-91. DOI: http://dx.doi.org/10 .1016/j.healthpol.2012.11.004.

4. Hopewell S, McDonald S, Clarke M, Egger M. Grey literature in meta- analyses of randomized trials of health care interventions. Cochrane Database of Systematic Reviews; 2008 2, MR000010.

5. Marzi C, Pardelli G, Sassi M. A terminology based re-definition of grey literature. Int J Grey Lit. 2011;7(1):19-23.

6. Boekhorst A, Farace D, Frantzen J. Grey literature survey 2004: a research project tracking developments in the field of grey literature. Presented at: Sixth International Conference on Grey Literature: Work on Grey in Progress; New York, NY; Dec 2004.

7. Emerald Insight. International Journal on Grey Literature [Internet]. Emerald Group Publishing; 2000 [cited 1 Jan 2014]. ,http://www.emeraldinsight .com/journals.htm?issn51466-6189&..

8. Farace DJ. Rise of the phoenix: a review of new forms and exploitations of gray literature. Pub Res Q. 1997; 13(2):69-76.

9. Schopfel J, Farace D. Grey literature. In: Bates MJ, Maack MN, eds. Encyclopedia of library and information sciences. 3rd ed. Taylor and Francis. New York, NY: CRC Press York; 2011. pp. 2029-39.

10. Artus HM. Old wine in new bottles? developments in electronic information and communication: structural change and functional inertia. Int J Grey Lit. 2005;1(1):9-16.

11. Pappas C, Williams I. Grey literature: its emerging importance. J Hosp Lib. 2011;11(3):228-34.

12. Benzies K, Premji S, Hayden KA, Serrett K. State-of-the-evidence reviews: advantages and challenges of including grey literature. Worldviews Evidence-Based Nurs. 2006;3:55-61.

13. Turner AM, Liddy ED, Bradley J, Wheatley JA. Modeling public health interventions for improved access to the gray literature. J Med Lib Assoc. 2005 Oct;93(4):487-94.

14. Luzi D. Trends and evolution in the development of grey literature: a review. Int J Grey Lit. 2000;1(3):106-17.

15. Higgins JPT, Green S, eds. Cochrane handbook for systematic reviews of interventions [Internet]. Version 5.1.0 Cochrane Collaboration; 2011 [cited 1 Jan 2014]. ,http://www .cochrane-handbook.org..

16. McAuley L, Pham B, Tugwell P, Moher D. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses? Lancet. 2000 Oct 7;356 (9237):1228-31.

17. Savoie I, Helmer D, Green CJ, Kazanjian A. Beyond MEDLINE: reducing bias through extended systematic review search. Int J Technol Assess Health Care. 2003 Winter;19(1):168-78.

18. Malina D, Nutt D. Grey literature is a feminist issue: women's knowledge and the Net. Int J Grey Lit. 2000; 1:18-27.

19. McKimmie T, Szurmak J. Beyond grey literature: how grey questions can drive research. J Agric Food Inf. 2002; 4(2):71-9.

20. Outten C. Gray literature [Internet]. California State University, Long Beach; 2012 [cited 1 Jan 2014]. ,http://csulb .libguides.com/print_content.php?pid 5120664&sid51038841&mode5g..

21. Gardner P, Fong A, Huang R. Measuring the impact of knowledge transfer from public research organisations: a comparison of metrics used around the world. Int J Learn Intellect Capital. 2010;7(3M):318-27.

22. Jacobs JA, Jones E, Gabella BA, Spring B, Brownson RC. Tools for implementing an evidence-based approach in public health practice. Prev Chronic Dis. 2012;9(11):1-9. DOI: http://dx.doi.org/10.5888/pcd9.110324.

23. Martin JLR, Pérez V, Sacrista M, Alvarez E. Is grey literature essential for a better control of publication bias in psychiatry? an example from three meta-analyses of schizophrenia. Eur Psychiatry. 2005 Dec;20(8):550-3.

24. Giustini D, Thompson D. Finding hard to finds: searching for grey literature [Internet]. University of British Columbia; 2006 [cited 13 Aug 2014]. Chttp:/ /blogs.ubc.ca/dean/files/2009/ 02/ greylit_manual_may 111 .pdf >.

25. Grey Literature International Steering Committee. Guidelines for the production of scientific and technical reports: how to write and distribute grey literature [Internet]. The Committee; 2006 [cited 1 Jan 2014]. ,http:// www.glisc.info/#2.3..

26. Denda K. Fugitive literature in the cross hairs: an examination of bibliographic control and access. Collection Manag. 2002;27(2):75-86.

27. Ranger S. Grey literature in special libraries: access and use. Publishing Res Q. 2005;21(1):53-63.

28. Mason MK. Grey literature: its history, definition, acquisition, and cataloguing [Internet]. 2013 [cited 1 Jan 2014]. ,http://www.moyak .com/papers / grey-technical-literature .html..

29. Sarli CC, Dubinsky EK, Holmes KL. Beyond citation analysis: a model for assessment of research impact. J Med Lib Assoc. 2010 Jan;98(1):17-23. http://dx.doi.org/10.3163/1536-5050 .98.1.008.

30. MacMillan HL, Wathen CN, Jamieson E, Boyle MH, Shannon HS, Ford-Gilboe M, Worster A, Lent B, Coben JH, Campbell JC, McNutt LA, McMaster Violence Against Women Research Group. Screening for intimate partner violence in health care settings: a randomized trial. JAMA. 2009 Aug 5;302(5):493-501.

31. Wathen CN, Macgregor JC, Sibbald SL, Macmillan HL. Exploring the uptake and framing of research evidence on universal screening for intimate partner violence against women: a knowledge translation case study. Health Res Policy Syst. 2013 Apr 12;11:13.

32. Bitsch V. Qualitative research: a grounded theory example and evaluation criteria. J Agribusiness. 2005;23(1):75-91.

33. Dembe AE, Lynch MS, Gugiu PC, Jackson RD. The translational research impact scale: development, construct validity, and reliability testing. Eval Health Prof. 2014 Mar;37(1):50-70.

34. Hanney SR, Gonzalez-Block MA, Buxton MJ, Kogan M. The utilisation of health research in policy-making: concepts, examples and methods of assessment. Health Res Policy Syst. 2003;1(1):2.

35. Jones TH, Donovan C, Hanney S. Tracing the wider impacts of biomedical research: a literature search to develop a novel citation categorisation technique. Scientometrics. 2012;93:125-34.

Received May 2014; accepted August 2014