Searching for Programme theories for a realist evaluation: a case study comparing an academic database search and a simple Google search

Abstract

Background

Realist methodologies are increasingly being used to evaluate complex interventions in health and social care. Programme theory (ideas and assumptions of how a particular intervention works) development is the first step in a realist evaluation or a realist synthesis, with literature reviews providing important evidence to support this. Deciding how to search for programme theories is challenging and there is limited guidance available. Using an example of identifying programme theories for a realist evaluation of Pressure Ulcer Risk Assessment Instruments in clinical practice, the authors explore and compare several different approaches to literature searching and highlight important methodological considerations for those embarking on a programme theory review.

Methods

We compared the performance of an academic database search with a simple Google search and developed an optimised search strategy for the identification primary references (i.e. documents providing the clearest examples of programme theories) associated with the use of Pressure Ulcer Risk Assessment Instruments (PU-RAIs). We identified the number of primary references and the total number of references retrieved per source. We then calculated the number needed to read (NNR) expressed as the total number of titles and abstracts screened to identify one relevant reference from each source.

Results

The academic database search (comprising CINAHL, The Cochrane Library, EMBASE, HMIC, Medline) identified 2 /10 primary references with a NNR of 1395.The Google search identified 7/10 primary references with a NNR of 10.1. The combined NNR was 286.3. The optimised search combining Google and CINAHL identified 10/10 primary references with a NNR of 40.2.

Conclusion

The striking difference between the efficiency of the review’s academic database and Google searches in finding relevant references prompted an in-depth comparison of the two types of search. The findings indicate the importance of including grey literature sources such as Google in this particular programme theory search, while acknowledging the need for transparency of methods. Further research is needed to facilitate improved guidance for programme theory searches to enhance practice in the realist field and to save researcher time and therefore resource.