Several libraries have asked us to address a pre-print of an article slated to be published in College & Research Libraries (C&RL) in July 2013 titled Paths of Discovery: Comparing the Search Effectiveness of EBSCO Discovery Service, Summon, Google Scholar, and Conventional Library Resources. The article is a well-intended effort by the authors to compare library discovery services, but the evaluation criteria used in the study inadvertently penalized the Summon service for many of the strongest features that have made it the market’s leading service. Further, the authors of the study clearly note that conclusions were drawn from data that was not statistically significant.
We believe the study would have been greatly improved by using a less subjective methodology and a wide array of other criteria should also have been considered including user experience and the breadth of each service’s content index. With these improvements we believe the outcome of this study would have been different.
When reviewing the study for yourself once it is published, we encourage you to ask the following critical questions:
The Summon unified index contains nearly 400 million newspaper articles from The New York Times, The Wall Street Journal, Washington Post, Los Angeles Times, Chicago Tribune, Financial Times, The Guardian, The Daily Telegraph, Jerusalem Post, South China Morning Post and other important publications. This is content that’s not easily discoverable in any other discovery service. For the many libraries that have deployed the Summon service, this is seen as a great strength. However, when students in the study’s Summon control group selected newspaper content for a given research task, the Summon service was severely penalized.
Of course, we understand that some libraries may have good reasons for de-emphasizing newspaper articles relative to other content. This is why the Summon service offers a myriad of ways to adjust holdings information or use facets to scope searches to easily exclude this content pre- or post-search. For instance, if only peer-reviewed content is valued, a peer-review filter powered by authoritative Ulrich’s data can be utilized to help users easily identify peer-reviewed materials. However, if the Summon service is not configured in a manner consistent with the study’s scoring system, it is obviously at a significant disadvantage.
Related to this, another factor that significantly influenced the study was that it involved a subjective evaluation of different students performing identical tasks but at different institutions, using different systems, searching different sets of resources, using different features, different scoping of results, different faceting options, and so forth. The authors readily acknowledge this issue when they state:
”…it is always prudent to use caution to avoid over-interpreting results based on subjective evaluations. Another potential limitation of this analysis involves the inherent difficulty in comparing students and research tools across universities, and in particular ascertaining whether students at the two universities were searching a corpus of research materials that is approximately equivalent.”
We believe the author’s warning that a subjective study should not be over-interpreted cannot be overemphasized.
Finally, it important to note that our continuing emphasis on usability as well as the ongoing investment we’ve made in expanding the Summon unified index to more than 1 billion items across more than 90 different content types has played a major part in making the Summon service the most prominent discovery solution in the market. The study did not consider either factor in its methodology. In our experience, both factors play a significant role when a library is making a decision about a discovery service and no study should be considered complete without taking them into account.
If you are currently evaluating discovery services, we strongly recommend that you consult with references and review case studies, presentations, ongoing usability studies and other reports that provide further insights into the remarkable impact that the Summon service is having on improving not only resource usage, but information literacy and library service models.