Evaluation and Comparison of Discovery Tools: An Update

Abstract

Selection and implementation of a web-scale discovery tool by the Rider University Libraries (RUL) in the 2011-2012 academic year revealed that the endeavor was a complex one. Research into the state of adoption of web-scale discovery tools in North America and the evolution of product effectiveness provided a good starting point. In the following study, we evaluated fourteen major discovery tools (three open source and ten proprietary), benchmarking sixteen criteria recognized as the advanced features of a "next generation catalog." Some of the features have been used in previous research on discovery tools. The purpose of the study was to evaluate and compare all the major discovery tools, and the findings serve to update librarians on the latest developments and user interfaces and to assist them in their adoption of a discovery tool.

Headnote

ABSTRACT

Selection and implementation of a web-scale discovery tool by the Rider University Libraries (RUL) in the 2011-2012 academic year revealed that the endeavor was a complex one. Research into the state of adoption of web-scale discovery tools in North America and the evolution of product effectiveness provided a good starting point. In the following study, we evaluated fourteen major discovery tools (three open source and ten proprietary), benchmarking sixteen criteria recognized as the advanced features of a "next generation catalog." Some of the features have been used in previous research on discovery tools. The purpose of the study was to evaluate and compare all the major discovery tools, and the findings serve to update librarians on the latest developments and user interfaces and to assist them in their adoption of a discovery tool.

INTRODUCTION

In 2004, the Rider University Libraries' (RUL) strategic planning process uncovered a need to investigate federated searching as a means to support research. A tool was needed to search and access all journal titles available to RUL users at that time, including 12,000+ electronic full-text journals. Lacking the ability to provide relevancy ranking due to its real-time search operations, as well as the cost of the products then available, the decision was made to defer implementation of federated search. Monitoring developments yearly revealed no improvements strong enough to adopt the approach. By 2011, the number of electronic full-text journals had increased to 51,128, and by this time federated search as a concept had metamorphosed into web -scale discovery. Clearly, the time had come to consider implementing this more advanced approach to searching the ever-growing number of journals available to our clients.

Though RUL passed on federated searching, viewing it as too cumbersome to serve our students well, we anticipated the day when improved systems would emerge. Vaughn nicely describes the ability of more highly evolved discovery systems to "provide quick and seamless discovery, delivery, and relevancy-ranking capabilities across a huge repository of content." 1 Yang and Hofmann anticipated the emergence of web-scale discovery with their evaluation of next generation catalogs. 2,3 By 2011, informed by Yang and Hofmann's research, we believed that the systems in the marketplace were sufficiently evolved to make our efforts at assessing available systems worthwhile. This coincided nicely with an important objective in our strategic plan : investigate link resolvers and discovery tools for federated searching and OPAC by summer 2011.

Heeding Alexander Pope's advice to "Be not the first by whom the new are tried, Nor yet the last to lay the old aside,"4 we set about discovering what systems were in use throughout North America and which features each provided.

SOME HISTORY

In 2006, Antelman, Lynema, and Pace observed that "library catalogs have represented stagnant technology for close to twenty years." Better technology was needed "to leverage the rich metadata trapped in the MARC record to enhance collection browsing. The promise of online catalogs has never been realized. For more than a decade, the profession either turned a blind eye to problems with the catalog or accepted that it is powerless to fix them." 6 Dissatisfaction with catalog search tools led us to review the VuFind Discovery Tool. While it had some useful features (spelling, "did you mean?" suggestions), it still suffered from inadequacies in full-text search and the cumbersome nature of searcher-designated Boolean searching. It did not work well in searching printed music collections and, of course, only served as a catalog front end.

With this all in mind, RUL developed a set of objectives to improve information access for clients:

* To provide information seekers with

* an easy search option for academically valid information materials

* an effective search option for academically valid information materials

* a reliable search option for academically valid information materials across platforms

* To recapture student academic search activity from Google

* To attempt revitalizing the use of monographic collections

* To provide an effective mechanism to support offerings of e -books

* To build a firm platform for appropriate library support of distance learning coursework

LITERATURE REVIEW

Marshall Breeding first discussed broad based discovery tools in 2005, shortly after the launch of Google Scholar. He posits that federated search could not compete with the power and speed of a tool like Google Scholar. He proclaims the need for, as he describes it, a "centralized search model."7

Building on Breeding's observations four years earlier, Diedrichs astutely observed in 2009 that "user expectations for complete and immediate discovery and delivery of information have been set by their experiences in the Web2.0 world. Libraries must respond to the needs of those users whose needs can easily be met with Google-like discovery tools, as well as those that require deeper access to our resources."10 In that same year, Dolski described the common situation in many academic libraries when in reference to the University of Nevada Las Vegas (UNLV) library he states, "Our library website serves as the de facto gateway to our electronic, networked content offerings. Yet usability studies have shown that findability, when given our website as a starting point, is poor. Undoubtedly this is due, at least in part, to interface fragmentation."11 This perfectly described the way we had come to view RUL's situation.

In 2010, Breeding reviewed the systems in the market, noting that these are not just next- generation catalogs. He stressed "equal access to content in all forms," a concept we now take for granted. A key virtue in discovery tools, he notes, is the "blending of the full text of journal articles and books alongside citation data, bibliographic, and authority records resulting in a powerful search experience. Rather than being provided a limited number of access points selected by catalogers, each word and phrase within the text becomes a possible point of retrieval." Breeding further points out that:

"web-scale discovery platforms will blur many of the restrictions and rules that we impose on library users. Rather than having to explain to a user that the library catalog lists books and journal titles but not journal articles, users can simply begin with the concept, author, or title of interest and straightaway begin seeing results across the many formats within the library's collection."12

Working with freshmen at Rider University revealed that they are ahead of the professionals in approaching information this way, and we believed that web-scale discovery tools could help our users.

As we began the process of selecting a discovery tool, we looked at the experiences of others. Fabbi at the University of Nevada Las Vegas (UNLV) folded in a strong component of organizational learning in a highly structured manner that was unnecessary at Rider.13 No information was disclosed on the process of selecting a discovery vendor, though the website reveals the presence of a discovery tool (http://library.nevada.edu/). In contrast, many librarians at Rider explored a variety of libraries' application of search tools. Following Hofmann and Yang's work, a process of vendor demonstrations and analysis of feasibility led to a trial of EBSCO Discovery Service.

What we hoped for is what Way at Grand Valley State reported in 2010 of his analysis of Serials Solutions' Summon:

An examination of usage statistics showed a dramatic decrease in the use of traditional abstracting and indexing databases and an equally dramatic increase in the use of full text resources from full text database and online journal collections. The author concludes that the increase in full text use is linked to the implementation of a web-scale discovery tool.14

METHOD

Understanding both RUL's objectives and the state of the art as reflected in the literature, we concluded that an up-to-date review of discovery tool adoptions was in order before moving forward in the process of selecting a product.

1. The resulting study included these steps:

(1) compiling a list of all the major discovery tools,

(2) developing a set of criteria for evaluation,

(3) examining between four to seven websites where a discovery tool was deployed and evaluating each tool against each criteria,

(4) Recording the findings, and

(5) analyzing the data.

The targeted population for the study included all the major discovery tools in use in the United States. We define a discovery tool as a library user interface independent of any library systems. A discovery tool can be used to replace the OPAC module of an integrated library system or live side- by-side with the OPAC. Other names for discovery tools include stand -alone OPAC, discovery layer, or discovery user interface. Lately, a discovery tool is more often called a discovery service because most are becoming subscription-based and reside remotely in a cloud-based SaaS (software as a service) model.

The authors compiled a list of fourteen discovery tools based on Marshall Breeding's "Major Discovery Products" guide published in "Library Technology Guides."15 Those included AquaBrowser Library, Axiell Arena, Bibli°Commons (Bibli°Core), Blacklight, EBSCO Discovery Service, Encore, Endeca, eXtensible Catalog, SirsiDynix Enterprise, Primo, Summon, Visualizer, VuFind, and Worldcat Local. Two open-source discovery layers, SOPAC (the Social OPAC) and Scriblio, were excluded from this study because very few libraries are using them.

For evaluation in this study, academic libraries were preferred over public libraries during the sample selection process. However, some discovery tools, such as Bibli°Commons, were more popular among public libraries. Therefore examples of public library websites were included in the evaluation. The sites that made the final list were chosen either from the vendor's website that maintained a customer list or Breeding's "Library Technology Guides ."16 The following is the final list of libraries whose implementations were used in the study.

Example Library Sites With Proprietary Discovery Tools:

AquaBrowser (Serials Solutions)

1. Allen County Public Library at http://smartcat.acpl.lib.in.us/

2. Gallaudet University Library at http://discovery.wrlc.org/?skin=ga

3. Harvard University at http://lib.harvard.edu/

4. Norwood Young America Public Library at http://aquabrowser.carverlib.org/

5. SELCO Southeastern Libraries Cooperating at http://aquabrowser.selco.info/?c_profile=far

6. University of Edinburgh (UK) at http://aquabrowser.lib.ed.ac.uk/

Axiell Arena (Axiell)

1. Doncaster Council Libraries (UK) at http://library.doncaster.gov.uk/web/arena

2. Lerums bibliotek (Lerums library, Sweden) at http://bibliotek.lerum.se/web/arena

3. London Libraries Consortium-Royal Kingston Library (UK) at http://arena.yourlondonlibrary.net/web/kingston

4. Norddjurs (Denmark) at https://norddjursbib.dk/web/arena/

5. North East Lincolnshire Libraries (UK) at http://library.nelincs.gov.uk/web/arena

6. Someron kaupunginkirjasto (Finland) at http://somero.verkkokirjasto.fi/web/arena

7. Syddjurs (Denmark) at https://bibliotek.syddjurs.dk/web/arena1

Bibli°Core (Bibli°Commons)

1. Halton Hills Public Library at http://hhpl.bibliocommons.com/dashboard

2. New York public Library at http://nypl.bibliocommons.com/

3. Oakville Public Library at http://www.opl.on.ca/

4. Princeton Public Library at http://princetonlibrary.bibliocommons.com/

5. Seattle Public Library at http://seattle.bibliocommons.com/

6. West Perth (Australia) Public Library at http://wppl.bibliocommons.com/dashboard

7. Whatcom County library System at http://wcls.bibliocommons.com/

EBSCO Discovery Service/EDS (EBSCO)

1. Aston University (UK) at http://www1.aston.ac.uk/library/

2. Columbia College Chicago Library at http://www.lib.colum.edu/

3. Loyalist College at http://www.loyalistlibrary.com/

4. Massey University (New Zealand) at http://www.massey.ac.nz/massey/research/library/library_home.cfm

5. Rider University at http://www.rider.edu/library

6. Santa Rosa Junior College at http://www.santarosa.edu/library/

7. St. Edward's University at http://library.stedwards.edu/

Encore (Innovative Interfaces)

1. Adelphi University at http://libraries.adelphi.edu/

2. Athens State University Library at http://www.athens.edu/library/

3. California State University at http://coast.library.csulb.edu/

4. Deakin University (Australia) at http://www.deakin.edu.au/library/

5. Indiana State University at http://timon.indstate.edu/iii/encore/home?lang=eng

6. Johnson And Wales University at http://library.uri.edu/

7. St. Lawrence University at http://www.stlawu.edu/library/

Endeca (Oracle)

1. John F. Kennedy Presidential Library and Museum at http://www.jfklibrary.org/

2. North Caroline State University at http://www.lib.ncsu.edu/endeca/

3. Phoenix Public Library at http://www.phoenixpubliclibrary.org/

4. Triangle Research Libraries Network at http://search.trln.org/

5. University of Technology, Sydney (Australia) at http://www.lib.uts.edu.au/

6. University of North Carolina at http://search.lib.unc.edu/

7. University of Ottawa (Canada) Libraries at http://www.biblio.uottawa.ca/html/index.jsp?lang=en

Enterprise (SirsiDynix)

1. Cerritos College at http://cert.ent.sirsi.net/client/cerritos

2. Maricopa County Community Colleges at https://mcccd.ent.sirsi.net/client/default

3. Mountain State University/University of Charleston at http://msul.ent.sirsi.net/client/default

4. University of Mary at http://cdak.ent.sirsi.net/client/uml

5. University of the Virgin Islands at http://uvi.ent.sirsi.net/client/default

6. Western Iowa Tech Community College at http://wiowa2.ent.sirsi.net/client/default

Primo (Ex Libris)

1. Aberystwyth University (UK) at http://primo.aber.ac.uk/

2. Coventry University (UK) at http://locate.coventry.ac.uk/

3. Curtin University (Australia) at http://catalogue.curtin.edu.au/

4. Emory University at http://web.library.emory.edu/

5. New York University at http://library.nyu.edu/

6. University of Iowa at http://www.lib.uiowa.edu/

7. Vanderbilt University at http://www.library.vanderbilt.edu

Visualizer (VTLS)

1. Blinn College at http://www.blinn.edu/Library/index.htm

2. Edward Via Virginia College of Osteopathic Medicine at http://vcom.vtls.com:1177/

3. George C. Marshall Foundation at http://gmarshall.vtls.com:6330/

4. Scugog Memorial Public Library at http://www.scugoglibrary.ca/

Summon (Serials Solutions)

1. Arizona State University at http://lib.asu.edu/

2. Dartmouth College at http://dartmouth.summon.serialssolutions.com/

3. Duke University at http://library.duke.edu/

4. Florida State University at http://www.lib.fsu.edu/

5. Liberty University at http://www.liberty.edu/index.cfm?PID=178

6. University of Sydney at http://www.library.usyd.edu.au/

Worldcat Local (OCLC)

1. Boise State University at http://library.boisestate.edu/

2. Bowie State University at http://www.bowiestate.edu/academics/library/

3. Eastern Washington University at http://www.ewu.edu/library.xml

4. Louisiana State University at http://lsulibraries.worldcat.org/

5. Saint John's University at http://www.csbsju.edu/Libraries.htm

6. Saint Xavier University at http://lib.sxu.edu/home

Examples of Open Source and Free Discovery Tools:

Blacklight (the University of Virginia Library)

1. Columbia University at http://academiccommons.columbia.edu/

2. Johns Hopkins University at https://catalyst.library.jhu.edu/

3. North Carolina University at http://historicalstate.lib.ncsu.edu

4. Northwestern University at http://findingaids.library.northwestern.edu/

5. Stanford University at http://www-sul.stanford.edu/

6. University Of Hull (UK) at http://blacklight.hull.ac.uk/

7. University of Virginia at http://search.lib.virginia.edu/

eXtensible Catalog/XC (eXtensible Catalog Organization/CARLI/University of Rochester)

1. Demo at http://extensiblecatalog.org/xc/demo

2. eXtensible Catalog Library at http://xco-demo.carli.illinois.edu/dtmilestone3

3. Kyushu University (Japan) at http://catalog.lib.kyushu-u.ac.jp/en

4. Spanish General State Authority Libraries (Spain) at http://pcu.bage.es/

5. Thailand Cyber University/Asia Institute of Technology (Thailand) at http://globe.thaicyberu.go.th/

VuFind (Villanova University)

1. Auburn University at http://www.lib.auburn.edu/

2. Carnegie Mellon University Libraries at http://search.library.cmu.edu/vufind/Search/Advanced

3. Colorado State University at http://lib.colostate.edu/

4. Saint Olaf College at http://www.stolaf.edu/library/index.cfm

5. University of Michigan at http://mirlyn.lib.umich.edu

6. Western Michigan University at https://catalog.library.wmich.edu/vufind/

7. Yale University Library at http://yufind.library.yale.edu/yufind/

The following list of criteria was used for the purpose of the evaluation. Some were based on those used by the previous studies on discovery tools.17, 18, 19 The list embodied the librarians' vision for the next-generation catalog and contained some of the most desirable features for a modern OPAC. The authors were aware of other desirable features for a discovery layer, and the following list was by no means the most comprehensive, but it served the purpose of the study well.

1. One-stop search for all library resources. A discovery tool should include all library resources in its search including the catalog with books and videos, journal articles in databases, and local archives and digital repository. This can be accomplished by the unified index or federated search, an essential component for a discovery tool. Some of the discovery tools are described as web-scale because of their potential to search seamlessly across all library resources.

2. State-of-the-art web interface. A discovery tool should have a modern design similar to e-commerce sites, such as Google, Netflix, and Amazon.

3. Enriched content. Discovery tools should include book cover images, reviews, and user- driven input, such as comments, descriptions, ratings, and tag clouds. The enriched content can be either from library patrons, commercial sources, or both.

4. Faceted navigation. Discovery tools should allow users to narrow down the search results by categories, also called facets. The commonly used facets include locations, publication dates, authors, formats, and more.

5. Simple keyword search box with a link to advanced search at the start page. A discovery tool should start with a simple keyword search box that looks like that of Google or Amazon. A link to the advanced search should be present.

6. Simple keyword search box on every page. The simple keyword search box should appear on every page of a discovery tool.

7. Relevancy. Relevancy results criteria should take into consideration circulation statistics and books with multiple copies. More frequently circulated books indicate popularity and usefulness, and they should be ranked higher on the top of the display. A book of multiple copies may also be an indication of importance.

8. "Did you mean . . . ? spell-checking. When an error appears in the search, the discovery tool should correct the query spelling as a link so that users can simply click on it to get the search results.

9. Recommendations/related materials. A discovery tool should recommend resources for readers in a similar manner to Amazon or other e -commerce sites, based on transaction logs. This should take the form of "readers who borrowed this item also borrowed the following . . . " or a link to recommended readings. It would be ideal if a discovery tool can recommend the most popular articles, a service similar to Ex Libris' bX Usage-based Services.

10. User contribution. User input includes descriptions, summaries, reviews, criticism, comments, rating and ranking, and tagging or folksonomies.

11. RSS feeds. A modern OPAC should provide RSS feeds.

12. Integration with social networking sites. When a discovery tool is integrated with social networking sites, patrons can share links to library items with their friends on social networks like Twitter, Facebook, and Delicious.

13. Persistent links. Records in a discovery tool contain a stable URL capable of being copied and pasted and serving as a permanent link to that record. They are also called permanent URLs.

14. Auto-completion/stemming. A discovery tool is equipped with the computational algorithm that it can auto-complete the search words or supply a list of previously used words or phrases for users to choose from. Google has stemming algorithms.

15. Mobile compatibility. There is a difference between being "mobile compatible" and a "custom mobile website." The former indicates a website can be viewed or used on a mobile phone, and the later denotes a different version of the user interface specially built for mobile use. In this study we include both as "yes."

16. Functional Requirements for Bibliographic Retrieval (FRBR). The latest development of RDA certainly makes a discovery tool more desirable if it can display FRBR relationships. For instance, a discovery tool may display and link different versions, editions or formats of a work, what FRBR refers to as expressions and manifestations.

For record keeping and analysis, a Microsoft Excel file with sixteen fields based on the above criteria was created. The authors checked the discovery tools on the websites of the selected libraries and recorded those features as present or absent.

RDA compatibility is not used as a criterion in the study because most discovery tools allow users to add RDA fields in MARC. By now, all the discovery tools should be able to display, index, and search the new RDA fields.

FINDINGS

One Stop searching for all library resources-This is the most desirable feature when acquiring a discovery tool. Unfortunately it also presented the biggest challenge for vendors. Both librarians and vendors have been struggling with this issue for the past several years, yet no one has worked out a perfect solution. Based on the examples the authors examined, this study found that only five out of fourteen discovery tools can retrieve articles from databases along with books, videos, and digital repositories. Those include EBSCO Discovery Service, Encore, Primo, Summon, and WorldCat Local. Whereas Encore uses an approach similar to federated search performing live searches of databases, the other discovery tools build a single unified index. While the single unified index requires the libraries to send their catalog data and local information to the vendor for update and thus the discovery tools may fall behind in reflecting up to the minute accuracy in local holdings, federated search does real-time searching and does not lag behind in displaying current information. Both approaches are limited in what they cover. Both need permission from content providers for inclusion in the unified index or to develop a connection to article databases for real-time searching.

For those discovery tools that do not have their own unified index or real-time searching capability, they provide web-scale searching through other means. For instance, VuFind has developed connectors to application programming interfaces (APIs) by Serials Solutions or OCLC to pull search results from Summon and Worldcat Local. Encore not only developed its own real- time connection to electronic databases but is enhancing its web-scale search by incorporating the unified index from other discovery tools such as the EBSCO Discovery service. AquaBrowser is augmented by 360 Federated Search for the same purpose. Despite of those possibilities, the authors did not find the article level retrieval in the sample discovery tools other than the main five mentioned above.

Comparing the coverage of each tools' web-scale index can be challenging. EBSCO, Summon, and WorldCat Local publicize their content coverage on the web while Primo and Encore only share this information with their customers. This makes it hard to compare and evaluate the content coverage without contacting vendors and asking for that information. At present, none of the five discovery tools (EBSCO Discovery Service, Encore, Primo, Summon, and WorldCat Local) can boast 100% coverage of all library resources. In fact, none of the Internet search engines, including Google or Google Scholar, can retrieve 100% of all resources. Therefore web-scale searching is more a goal than a possibility. Apart from political and economic reasons, this is in part due to the nonbibliographic structure of the contents in databases such as SciFinder and some others. One stop searching is still a work in progress because discovery tools provide students with a quick and simple way to retrieve a large number, but still an incomplete list of resources held by a library. For more in-depth research, students are still encouraged to search the catalog, discipline specific databases, and digital repositories separately.

State of the art interface-All the discovery tools are very similar in appearance to amazon.com. Some are better than others. This study did not rate each discovery tool based on a scale and thus did not distinguish their fine degrees in appearance. Rather each discovery tool is given a "Yes" or "No." The designation was based on subjective judgment. All the discovery tools received "Yes" because they are very similar in appearance.

Enriched content-All the discovery tools have embedded book cover images or video jacket images, but some have displayed more, such as ratings and rankings, user-supplied or commercially available reviews, overviews, previews, comments, descriptions, title discussion, excerpts, or age suitability, just to name a few. A discovery tool may display enriched content by default out of box, but some may need to be customized to include it. The following is a list of enriched content implemented in each discovery tool that the authors found in the sample. The number in the last column indicates how many types of enriched content were found in the discovery tool at the time of the study. Bibli°Commons and AquaBrowser stand out from the rest and made the top two on the list based on the number of enriched content from noncataloging sources (see figure 1). It is debatable how much nontraditional data a discovery tool should incorporate into its display. It warrants another discussion as to how useful such data is for users.

Faceted navigation-Faceted navigation has become a standard feature in discovery tools over the last two years. It allows users to further divide search results into subsets based on pre- determined terms. Facets come from a variety of fields in MARC records. Some discovery tools have more facets than others. The most commonly seen facets include location or collections, publication dates, formats, author, genre, and subjects. Faceted navigation is highly configurable as many discovery tools allow libraries to decide on their own facets. Faceted navigation has become an integral part of a discovery tool.

Simple keyword search box on the starting page with a link to advanced search-The original idea is to allow a library's user interface to resemble Google by displaying a simple keyword search box with a link to advanced search at the starting page. Most discovery tools provide the flexibility for libraries to choose or reject this option. However, many librarians find this approach unacceptable as they feel it lacks precision in searching and thus may mislead users. As the keyword box is highly configurable and up to the library to decide how they will present it, many libraries have added a pull down menu with options to search keywords, authors, titles, and locations. In doing so, the original intention for a Google like simple search box is lost. Therefore only a few libraries follow the Google-like box style at the starting page. Most libraries altered the simple keyword search box on the starting page to include a dropdown menu or radio buttons, so the simple keyword search box is neither simple nor limited to keyword search only. Nevertheless, this study gave all the discovery tools a "Yes." All the systems are capable of this feature even though libraries may choose not to use it.

Simple keyword search box on every page-The feature enables a user to start a new search at every step of navigation in the discovery tool. Most of the discovery tools provide such a box on the top of the screen as users navigate through the search results and record displays except eXtensible Catalog and Enterprise by SirsiDynix. The feature is missing from the former while the latter almost has this feature except when displaying bib records in a pop-up box.

Relevancy-Traditionally, relevancy is uniformly based on a computer algorithm that calculates the frequency and relative position of a keyword (field weighting) in a record and displays the search results based on the final score. Other factors have never been a part of the decision in the display of search results. In the discussion on next-generation catalogs, relevancy based on circulation statistics and other factors came up as a desirable possibility, and no discovery tool has met this challenge until now. Primo by Ex Libris is the only one among the discovery tools under investigation that can sort the final results by popularity. "Primo's popularity ranking is calculated by use. This means that the more an item record has been clicked and viewed, the more popular it is."20 Even though those are not real circulation statistics, this is considered to be a revolutionary step and a departure from traditional relevancy. Three years ago none of the discovery tools provided this option.21 To make relevancy ranking even more sophisticated, ScholarRank, another service by Ex Libris, can work with Primo to sort the search results not only based on a query match but also an item's value score (its usage and number of citations) and a user's characteristics and information needs. This shows the possibility of more advanced relevancy ranking in discovery tools. Other vendors will most likely follow in the future incorporating more sophistication in their relevancy algorithms.

Spell checker/"Did you mean . . . ?"-The most commonly observed way of correcting a misspelling in a query is, "Did you mean . . . ?" but there are other variations providing the same or similar services. Some of those variations are very user-friendly. The following is a list of different responses when a user enters misspelled words (see figure 2). "xxx" represents the keyword being searched.

Most of the discovery tools on the list provide this feature except Blacklight and eXtensible Catalog. Open-source solutions sometimes provide a framework that you add features to. This leaves many possibilities for local developers to add and develop. For instance, a dictionary or spell checker may be easily installed even if a discovery tool does not come with one out of the box. This feature may be configurable.

9. Recommendation-Amazon has one of those search engines with a recommendation system such as "customers who bought item A also bought item B." The ecommerce recommendation algorithms analyze the activities of shoppers on the web and build a database of buyer profiles. The recommendations are made based on shopper behavior. When this applies to the library content, it could become "readers who were interested in item A were also interested in item B ." However, most discovery tools do not have a recommendation system. Instead, they have adopted different approaches. Most discovery tools make recommendations from bibliographic data in MARC records such as subject headings for similar items. Primo is one of the few discovery tools with a recommendation system similar to those used by Amazon and other Internet commercial sites. Its bX Article Recommender Service is based on usage patterns collected from its link resolver, SFX. Developed by Ex Libris, bX is an independent service that integrates with Primo well, but can serve as an add-on function for other discovery tools. bX is an excellent example that discovery tools can suggest new leads and directions for scholars in their research. The authors counted all the discovery tools that provide some kind of recommendations regardless of their technological approaches using MARC data or algorithms. Ten out of fourteen discovery tools provide this feature in various forms (see figure 3). Those include Axiell Arena, Bibli°Commons, EBSCO Discovery Service, Encore, Endeca, eXtensible Catalog, Primo, Summon, WorldCat Local, and VuFind. The following are some of the recommendations found in those discovery tools. The authors did not find any recommendation in the libraries that use AquaBrowser, Enterpri se, Visualizer, or Blacklight.

Some discovery tool recommendations are designed in a more user friendly manner than others. Most recommendations exist exclusively for items. Ideally, a discovery tool should provide an article recommendation system like Ex Libris ' bX Usage-based Service that will show users the most frequently used and most popular articles. At the time of this evaluation, no discovery tool has incorporated an article recommendation system except Primo. Research is needed to evaluate how patrons utilize recommendation services or if they find recommendations beneficial in discovery tools.

User contribution-Traditionally, bibliographic data has been safely guarded by cataloging librarians for quality control. It has been unthinkable that users would be allowed to add data to library records. The Internet has brought new perspectives on this issue. Half of the discovery tools (7) under evaluation provide this feature to varying degrees (see figure 4). Designed primarily for public libraries, Bibli°Commons seems the most open to user-supplied data among all the discovery tools. Many other discovery tools (7) allow users to contribute tags and reviews. All the discovery tools allow librarians to censor user-supplied data before releasing it for public display. The following figure is a summary of the types of data these discovery tools allow users to enter.

Past research indicates that folksonomies or tags are highly useful.22 They complement library- controlled vocabularies, such as Library of Congress Subject Headings, and increase access to library collections. A few discovery tools allow user entered tags to form "word clouds." The relative importance of tags in a word cloud is emphasized by font color and size. A tag list is another way to organize and display tags. In both cases, tags are hyperlinked to a relevant list of items. Some tags serve as keywords to start new searches, while others narrow search results. Only four discovery tools, AquaBrowser, Encore, Primo, and WorldCat Local, provide both tag clouds and lists. Bibli°Commons provides only tag lists for the same purpose. The rest of the discovery tools do not have either. One setback of user-supplied tags for subject access is their incomplete nature. They may lead users to partial retrieval of information as users add tags only to items that they have used. The coverage is not systematic and inclusive of all collections. Therefore data supplied by users in discovery tools remains controversial. It is possible to seed systems with folksonomies using services like LibraryThing for Libraries, which could reduce the impact of this issue.

RSS Feed/email Alerts-This feature can automatically send a list of new library resources to users based on his or her search criteria. It can be useful for experienced researchers or frequent library users. Some discovery tools may use email alerts as well. Eight out of fourteen discovery tools in this evaluation provide RSS feeds. Those with RSS feeds include AquaBrowser, Axiell Arena, EBSCO Discovery Service, Endeca, Enterprise, Primo, Summon, and VuFind. An RSS feed can be added as a plug-in in some discovery tools if it does not come as part of the base system.

Integration with social networking sites-As most of the college students participate in social networking sites, this feature provides an easy way to share resources among college students on social networking sites. Users can place the link to a resource by clicking on an icon in the discovery tool and share the resource with friends on Facebook, Twitter, Delicious and many other social network sites. Nine out of the fourteen discovery tools provide this feature. Some discovery tools provide integration possibilities with many more social networking sites than others. Those with this feature include AquaBrowser, Axiell Arena, Bibli°Commons, EBSCO Discovery Service, Encore, Endeca, Primo, WorldCat Local, and eXtensible Catalog. So far, the interaction between discovery tools and social networking sites is limited to sharing resources. Social networking sites should be carefully evaluated for the possibility of integrating some of their popular features into discovery tools.

Persistent link-This is also called permanent link or permURL. Not all the links displayed in a browser location box are persistent links, therefore some discovery tools specifically provide a link in the records for users to copy and keep. Five out of fourteen discovery tools explicitly listed this link in records. Those include AquaBrowser, Axiell Arena, Blacklight, EBSCO Discovery Service, and WorldCat Local. The authors marked a system as "No" when a permanent link is not prominently displayed in a discovery tool. In other words, only those discovery tools that explicitly provide a persistent link are counted as "Yes." However, the URL in a browser 's location box during the display of a record may serve as a persistent link in some cases. For instance, VuFind does not provide a permanent URL in the record, but indicates on the project site that URL in the location box is a persistent link.

Auto-completion/stemming-When a user types in keywords in the search box, the discovery tool will supply a list of words or phrases that she or he can choose readily. This is a highly useful feature that Google excels at. Stemming not only automatically completes the spelling of a keyword, but also supplies a list of phrases that point to existing items. The authors found this feature in six out of fourteen discovery tools. They include Axiell Arena, Endeca, Enterprise, Extensible Catalog, Summon, and WorldCat Local.

Mobile Interface-The terms "mobile compatible" and "mobile interface" are two different concepts. A mobile interface is a simplified version of a normal browser version of a discovery tool interface so it is optimized for use on mobile phones, and the authors only counted those discovery tools that have a separate mobile interface. A discovery tool may be mobile friendly or compatible and does not necessarily need a separate mobile interface. Many discovery tools, such as EBSCO, can detect the request from a mobile phone and automatically direct the request to the mobile interface. Eleven out of fourteen claim to provide a separate mobile interface. Blacklight, Enterprise, and eXtensible Catalog do not seem to have a separate mobile interface even though they may be mobile friendly.

FRBR-FRBR groupings denote the relationships between work, manifestation, expression, and items. For instance, a search will not only retrieve a title, but different editions and formats of the work. Only three discovery tools can display FRBR relationships: eXtensible Catalog (open source), Primo by Ex Libris, and Worldcat Local by OCLC. So far, most discovery tools are not capable of displaying the manifestations and expressions of a work in a meaningful way. From the user's point of view, this feature is highly desirable. Figure 5 is a screenshot from Primo demonstrating displays indicating a large number of different adaptations of the work "Romeo and Juliet." Figure 6 displays the same intellectual work in different manifestations such as DVD, VHS, books, and more.

SUMMARY

The following are the summary tables of our comparison and evaluation. Proprietary and open source programs are listed separately in these tables. The total number of features the authors found in a particular discovery tool is displayed at the end of the column. Proprietary discovery tools seem to have more advanced characteristics of a modern discovery tool than the open- source counterparts. The open-source program Blacklight displays fewer advanced features, but seems flexible for users to add features. See figures 7, 8, and 9.

As one-stop searching is the core of a discovery tool, this consideration placed five discovery tools above the rest: Encore, EBSCO Discovery Service, Primo, Summon, and WorldCat Local (see figure 10). These five are web-scale discovery services. All of them use their native unified index except Encore, which has incorporated the EBSCO Unified Index in its search. Despite of great progress made in the past three years in one-stop searching, none of the discovery tools can truly search across all library resources-all of them have some limitations as to the coverage of content. Each unified index may cover different databases as well as overlap each other in many areas. One possible solution may lie in a hybrid approach that combines a unified index with federated search (also called real-time discovery). Those old and new technologies may work well when complementing each other. It remains a challenge if libraries will ever have one-stop searching in its true sense.

It is also worth mentioning that one-stop searching is a vital and central piece of discovery tools. Those discovery tools without a native unified index or connectors to databases for real -time searching are at a disadvantage. Therefore discovery tools that do not provide web-scale searching are investigating various possibilities to incorporate one-stop searching. Some are drawing on the unified indexes of those discovery tools that have them through connectors to the application programming interfaces (APIs) of those products. For instance, VuFind includes connectors to the APIs of a few other systems that have a unified index or vast resources such as Summon and Worldcat. Blacklight may provide one-stop searching through the Primo API. Such a practice may present other problems such as calculating relevancy ranking across resources that may not live in the same centralized index, thus not achieving fully balanced relevancy ranking. Nevertheless, discovery tool developers are working hard to achieve one-stop searching. As a unified index can be shared across discovery tools, in the next few years, more and more discovery services may offer one-stop searching.

Based on the count of the sixteen criteria in the checklist, we ranked Primo and WorldCat Local as the top two discovery tools. Based on our criteria, Primo has two unique features that make it stand out: relevancy enhanced by usage statistics and value score and the FRBR relationship display. WorldCat Local and Extensible Catalog are the other two discovery tools that can display FRBR relationships (see figure 11).

LIMITATIONS

As discovery tools are going through new releases and improvements, what is true today may be false tomorrow. Discovery tools constantly improve and evolve, and many features are not included in this evaluation, such as integration with Google Maps for the location of an item and user-driven acquisitions. Innovations are added to discovery tools constantly. This study only covers the most common features that the library community agreed upon as those that a discovery tool should have.

Some open-source discovery tools may provide a skeleton of an application that leaves the code open for users to develop new features. Therefore different implementations of an open-source discovery tool may encompass totally different features that are not part of the core application. For instance, the University of Virginia developed Virgo based on Blacklight, adding many advanced features. Thus it is quite a challenge to distinguish what comes with the software and what are local developments.

This study focused on the user interface of discovery tools. What are not included are content coverage, application administration, and searching capability of the discovery tools. Those three are important factors when choosing a discovery tool.

CONCLUSION

Search technology has evolved far beyond federated searching. The concept of a "Next Generation Catalog" has merged with this idea, and spawned a generation of discovery tools bringing almost Google-like power to library searching. The problems facing libraries now are the intelligent selection of a tool that fits their contexts, and structuring a process to adopt and refine that tool to meet the objectives of the library. Our findings indicate that Primo and WorldCat Local have better user interfaces, displaying more advanced features of a Next Generation Catalog than their peers.

For RUL, EBSCO Discovery Service (EDS) provides something approaching the ease of Google searching from either a single search box or a very powerful advanced search. Being aware of the limitations noted above, Rider's libraries elected to continue displaying traditional search options in addition to what we've branded "Library One Search." Another issue we discovered in this process is when negotiating for a vendor-hosted test, libraries must be sure that the test period begins when the configuration is complete rather than only when the data load begins. All phases of the project took far more time than anticipated. The client institution's implementation coordinator or team needs to be reviewing the progress on a daily basis and communicating often with the vendor-based implementation team. With the evaluative framework this study provides, libraries moving toward discovery tools should consider changing capabilities of the available discovery tools to make informed choices.

References

REFERENCES

1. Jason Vaughan, "Investigations into Libra ry Web-Scale Discovery Services," Information Technology & Libraries 31, no. 1 (2012): 32-33, http://dx.doi.org/10.6017/ital.v31i1.1916.

2. Sharon Q. Yang and Melissa A. Hofmann, "Next Generation or Current Generation? A Study of the OPACs of 260 Academic Libraries in the USA and Canada," Library Hi Tech 29 no. 2 (2011): 266-300.

3. Melissa A. Hofmann and Sharon Q. Yang, "'Discovering' What's Changed: A Revisit of the OPACs of 260 Academic Libraries," Library Hi Tech 30, no. 2 (2012): 253-74.

4. Alexander Pope, "Alexander Pope Quotes," http://www.brainyquote.com/quotes/authors/a/alexander_pope.html.

5. F. William Chickering, "Linking Information Technologies: Benefits and Challenges," Proceedings of the 4th International Conference on New Information Technologies, Budapest, Hungary, December 1991, http://web.simmons.edu/~chen/nit/NIT%2791/019-chi.htm.

6. Kristin Antelman, Emily Lynema, and Andrew K. Pace, "Toward a Twenty-First Century Library Catalog," Information Technology & Libraries 25, no. 3, (2006): 128-39, http://dx.doi.org/10.6017/ital.v25i3.3342.

7. Marshall Breeding, "Plotting a New Course for Metasearch," Computers in Libraries 25, no. 2 (2005): 27-29.

8. Judith Carter, "Discovery: What Do You Mean by That?" Information Technology & Libraries 28, no. 4 (2009): 161-63, http://dx.doi.org/10:6017/ital.v28i4.3326.

9. Priscilla Caplan, "On Discovery Tools, OPACs and the Motion of Library Language," Library Hi Tech 30, no. 1 (2012): 108-15.

10. Carol Pitts Diedrichs, "Discovery and Delivery: Making it Work for Users," Serials Librarian 56, no. 1-4 (2009): 79, http://dx.doi.org/10.1080/03615260802679127.

11. Alex A. Dolski, "Information Discovery Insights Gained from MultiPAC, a Prototype Library Discovery System," Information Technology & Libraries 28, no. 4, (2009): 173, http://dx.doi.org/10.6017/ital.v28i4.3328.

12. Marshall Breeding, "The State of the Art in Library Discovery," Computers in Libraries 30, no. 1 (2010): 31-34.

13. Jennifer L. Fabbi, "Focus as Impetus for Organizational Learning," Information Technology & Libraries 28, no. 4 (2009): 164-71, http://dx.doi.org/10.6017/ital.v28i4.3327.

14. Douglas Way, "The Impact of Web-scale Discovery on the Use of a Library Collection," Serials Review 36, no. 4: (2010): 214-20, http://dx.doi.org/10.1016/j.serrev.2010.07.002.

15. Marshall Breeding, "Library Technology Guides: Discovery Products," http://www.librarytechnology.org/discovery.pl.

16. Ibid.

17. Sharon Q. Yang and Kurt Wagner, "Evaluating and Comparing Discovery Tools: How Close Are We towards Next Generation Catalog?" Library Hi Tech 28, no. 4 (2010): 690-709.

18. Yang and Hofmann, "Next Generation or Current Generation?" 266-300.

19. Melissa A. Hofmann and Sharon Q. Yang, "How Next-Gen R U? A Review of Academic OPACS in the United States and Canada," Computers in Libraries 31, no. 6 (2010): 26-29.

20. Brown Library of Virginia Western Community College, "Primo-Frequently Asked Questions," http://www.virginiawestern.edu/library/primo-faq.php#popularity_ranking.

21. Yang and Wagner, "Evaluating and Comparing Discovery Tools," 690-709.

22. Yanyi Lee and Sharon Q. Yang, "Folksonomies as Subject Access-A Survey of Tagging in Library Online Catalogs and Discovery Layers," paper presented at IFLA post-conference "Beyond Libraries-Subject Metadata in the Digital Environment and Semantic Web," Tallinn, EstoniaI, 18 August 2012, http://www.nlib.ee/html/yritus/ifla_jarel/papers/4-1_Yan.docx