Usability Test Results for a Discovery Tool in an Academic Library

Abstract

Discovery tools are emerging in libraries. These tools offer library patrons the ability to concurrently search the library catalog and journal articles. While vendors rush to provide feature-rich interfaces and access to as much content as possible, librarians wonder about the usefulness of these tools to library patrons. To learn about both the utility and usability of EBSCO Discovery Service, James Madison University (JMU) conducted a usability test with eight students and two faculty members. The test consisted of nine tasks focused on common patron requests or related to the utility of specific discovery tool features. Software recorded participants' actions and time on task, human observers judged the success of each task, and a post-survey questionnaire gathered qualitative feedback and comments from the participants. Participants were successful at most tasks, but specific usability problems suggested some interface changes for both EBSCO Discovery Service and JMU's customizations of the tool. The study also raised several questions for libraries above and beyond any specific discovery-tool interface, including the scope and purpose of a discovery tool versus other library systems, working with the large result sets made possible by discovery tools, and navigation between the tool and other library services and resources. This article will be of interest to those who are investigating discovery tools, selecting products, integrating discovery tools into a library web presence, or performing evaluations of similar systems. [PUBLICATION ABSTRACT]

Headnote

ABSTRACT

Discovery tools are emerging in libraries. These tools offer library patrons the ability to concurrently search the library catalog and journal articles. While vendors rush to provide feature-rich interfaces and access to as much content as possible, librarians wonder about the usefulness of these tools to library patrons. To learn about both the utility and usability of EBSCO Discovery Service, James Madison University (JMU) conducted a usability test with eight students and two faculty members. The test consisted of nine tasks focused on common patron requests or related to the utility of specific discovery tool features. Software recorded participants' actions and time on task, human observers judged the success of each task, and a post-survey questionnaire gathered qualitative feedback and comments from the participants. Participants were successful at most tasks, but specific usability problems suggested some interface changes for both EBSCO Discovery Service and JMU's customizations of the tool. The study also raised several questions for libraries above and beyond any specific discovery-tool interface, including the scope and purpose of a discovery tool versus other library systems, working with the large result sets made possible by discovery tools, and navigation between the tool and other library services and resources. This article will be of interest to those who are investigating discovery tools, selecting products, integrating discovery tools into a library web presence, or performing evaluations of similar systems.

INTRODUCTION

Discovery tools appeared on the library scene shortly after the arrival of next-generation catalogs. The authors of this paper define discovery tools as web software that searches journal-article and library-catalog metadata in a unified index and presents search results in a single interface. This differs from federated search software, which searches multiple databases and aggregates the results. Examples of discovery tools include Serials Solutions Summon, EBSCO Discovery Service, Ex Libris Primo, and OCLC WorldCat Local; examples of federated search software include Serials Solutions WebFeat and EBSCO Integrated Search. With federated search software, results rely on the search algorithm and relevance ranking as well as each tool's algorithms and relevance rankings.

Discovery tools, which import metadata into one index, apply one set of search algorithms to retrieve and rank results. This difference is important because it contributes to a fundamentally different user experience in terms of speed, relevance, and ability to interact consistently with results. Combining the library catalog, article indexes, and other source types in a unified interface is a big change for users because they no longer need to choose a specific search tool to begin their search. Research has shown that such a choice has long been in conflict with users' expectations.1 Federated search software was unable to completely fulfill users' expectations because of its limited technology.2 Now that discovery tools provide a truly integrated search experience, with greatly improved relevance rankings, response times, and increased consistency, libraries can finally begin to meet this area of user expectation. However, discovery tools present new challenges for users: will they be able to differentiate between source types in the integrated results sets? Will they be able to limit large results sets effectively? Do they understand the scope of the tool and that other online resources exist outside the tool's boundaries?

The sea change brought by discovery tools also raises challenges for librarians, who have grown comfortable with the separation between the library catalog and other online databases. Discovery tools may mask important differences between disciplinary searching, and they do not currently offer discipline-specific strategies or limits. They also lack authority control, which makes topical precision a challenge. Their usual prominence on library websites may direct traffic away from carefully cultivated and organized collections of online resources. Discovery tools offer both opportunities and challenges for library instruction, depending on the academic discipline, users' knowledge, and information-seeking need.

James Madison University (JMU) is a predominantly undergraduate institution of approximately 18,000 students in Virginia. JMU has a strong information literacy program integrated into the curriculum through the university's Information Seeking Skills Test (ISST). The ISST is completed before students are able to register for third-semester courses. Additionally, the library provides an information literacy tutorial, "Go for the Gold," that supports the skills needed for the ISST.

JMU launched EBSCO Discovery Service (EDS) in August 2010 after participating as a beta development partner in spring and summer 2010. As with other discovery tools, the predominant feature of EDS is integration of the library catalog with article databases and other types of sources. At the time of this study, EDS had a few differentiating features. First, because of EBSCO's business as a database and journal provider, article metadata was drawn from a combination of journal-publisher information and abstracts and index records. The latter included robust subject indexing (e.g., the medical subject headings in CINAHL). The content searched by EDS varies by institution according to the institution's subscription. JMU had a large number of EBSCO databases and third-party database subscriptions through EBSCO, so the quantity of information searched by EDS at JMU is quite large.

EDS also allowed for extensive customization of the tool, including header navigation links, results-screen layout, and the inclusion of widgets in the right-hand column of the results screen. JMU Libraries developed a custom "Quick Search" widget based on EDS for the library home page (see figure 1), which allows users to add limits to the discovery-tool search and assists with local authentication requirements. Based on experience with a pilot test of the open-source VuFind next-generation catalog, JMU Libraries believed users would find the ability to limit up-front useful, so Quick Search's first drop-down menu contained keyword, title, and author field limits; the second drop-down contained limits for books, articles, scholarly articles, "Just LEO Library Catalog," and the library website (which did not use EDS). The "Just LEO Library Catalog" option limited the user's search to the library catalog database records but used the EDS interface to perform the search. To access the native catalog interface, a link to LEO Library Catalog was included immediately above the search box as well as in the library website header.

Evaluation was included as part of the implementation process for the discovery tool, and therefore a usability test was conducted in October 2010. The purpose of the study was to explore how patrons used the discovery tool, to uncover any usability issues with the chosen system and to investigate user satisfaction. Specific tasks addressed the use of facets within the discovery tool, patrons' use of date limiters, and the usability of the Quick Search widget. The usability test also had tasks in which users were asked to locate books and articles using only the discovery tool, then repeat the task using anything but the discovery tool.

This article interprets the usability study's results in the context of other local usability tests and web-usage data from the first semester of use. Some findings were used to implement changes to Quick Search and the library website, and to recommend changes to EBSCO; however, other findings suggested general questions related to discovery tool software that libraries will need to investigate further.

LITERATURE REVIEW

Literature reviewed for this article included some background reading on users and library catalogs, library responses to users' expectations, usability studies in libraries, and usability studies of discovery tools specifically.

The first group of articles comprised a discussion about the limitations of traditional library catalogs. The strengths and weaknesses of library catalogs were reported in several academic libraries' usability studies.3 Calhoun recognized that library users' preference for Google caused a decline in the use and value of library catalogs, and encouraged library leaders to "establish the catalog within the framework of online information discovery systems." 4 This awareness of changes in user expectations during a time when Google set the benchmark for search simplicity was echoed by numerous authors who recognized the limits of library catalogs and expressed a need for the catalog to be greatly modernized to keep pace with the evolution of the web.5

Libraries have responded in several ways to the call for modernization, most notably through investigations related to federated searching and next-generation catalogs. Several articles have presented usability studies results for various federated searching products.6 Fagan provided a thorough literature review of faceted browsing and next-generation catalogs.7 Western Michigan University presented usability study results for the next-generation catalog VuFind, revealing that participants took advantage of the simple search box but did not use the next-generation catalog features of tagging, comments, favorites, and SMS texting.8 The University of Minnesota conducted two usability studies of Primo and reported that participants were satisfied with using Primo to find known print items, limit by author and date, and find a journal title.9 Tod Olson conducted a study with graduate students and faculty using the AquaBrowser interface, and his participants located sources for their research they had not previously been able to find.10

The literature also revealed both opportunities and limitations of federated searching and next-generation catalogs. Allison presented statistics from Google Analytics for an implementation of Encore at the University of Nebraska-Lincoln. 11 The usage statistics revealed an increased use of article databases as well as an increased use of narrowing facets such as format and media type, and library location. Allison concluded that Encore increased users' exposure to the entire collection. Breeding concluded that federated searching had various limitations, especially search speed and interface design, and was thus unable to compete with Google Scholar.12 Usability studies of next-generation catalogs revealed a lack of features necessary to fully incorporate an entire library's collection. Breeding also recognized the limitations of next-generation library catalogs and saw discovery tools as their next step in evolution: "It's all about helping users discover library content in all formats, regardless of whether it resides within the physical library or among its collections of electronic content, spanning both locally owned materials and those accessed remotely through subscriptions." 13

The dominant literature related to discovery tools discussed features,14 reviewed them from a library selector perspective,15 summarized academic libraries' decisions following selection,16 presented questions related to evaluation after selection,17 and offered a thorough evaluation of common features.18 Allison concluded that "usability testing will help clarify what aspects need improvement, what additions will make [the interface] more useful, and how the interface can be made so intuitive that user training is not needed."19 Breeding noted "it will only be through the experience of library users that these products will either prove themselves or not."20

Libraries have been adapting techniques from the field of usability testing for over a decade to learn more about user behavior, usability, and user satisfaction, with library web sites and systems. 21 Rubin and Chisnell and Dumas and Redish provided an authoritative overview of the benefits and best practices of usability testing. 22 In addition, Campbell and Norlin and Winters offered specific usability methodologies for libraries.23

WorldCat Local has dominated usability studies of discovery tools published to date. Ward, Shadle, and Mofield conducted a usability study at the University of Washington.24 Although the second round of testing was not published, the first round involved seven undergraduate and three graduate students; its purpose "was to determine how successful UW students would be in using WorldCat Local to discover and obtain books and journal articles (in both print and electronic form) from the UW collection, from the Summit consortium, and from other WorldCat libraries."25 Although participants were successful at completing these tasks, a few issues arose out of the usability study. Users had difficulty with the brief item display because reviews were listed higher than the actual items. The detailed item display also hindered users' ability to decipher between various editions and formats. The second round of usability testing, not yet published, included tasks related to finding materials on specific subject areas.

Boock, Chadwell, and Reese conducted a usability study of WorldCat Local at Oregon State University.26 The study included four tasks and five evaluative questions. Forty undergraduate students, sixteen graduate students, twenty-four library employees, four instructors, and eighteen faculty members took part in the study. They summarized that users found known-title searching to be easier in the library catalog but found topical searches to be more effective in WorldCat Local.The participants preferred WorldCat Local for the ability to find articles and search for materials in other institutions.

Western Washington University also conducted a usability study of WorldCat Local. They selected twenty-four participants with a wide range of academic experience to conduct twenty tasks in both WorldCat Local and the traditional library catalog.27 The comparison revealed several problems in using WorldCat Local, including users' inability to determine the scope of the content, confusion over the intermixing of formats, problems with the display of facet option, and difficulty with known-item searches. Western Washington University decided not to implement WorldCat Local.

OCLC published a thorough summary of several usability studies conducted mostly with academic libraries piloting the tool, including the University of Washington; the University of California (Berkeley, Davis, and Irvine campuses); Ohio State University; the Peninsula Library System in San Mateo, California; and the Free Library of Urbana and the Des Plaines Public Library, both in Illinois.28 The report conveyed favorable user interest in searching local, group, and global collections together. Users also appreciated the ability to search articles and books together. The authors commented, "however, most academic participants in one test (nine of fourteen) wrongly assumed that journal article coverage includes all the licensed content available at their campuses."29 OCLC used the testing results to improve the order of search results, provide clarity about various editions, improve facets for narrowing a search, provide links to electronic resources, and increase visibility of search terms.

At Grand Valley State University, Doug Way conducted an analysis of usage statistics after implementing the discovery tool Summon in 2009; the usage statistics revealed an increased use of full-text downloads and link resolver software but a decrease in the use of core subject databases.30 The usage statistics showed promising results, but Way recommended further studies of usage statistics over a longer period of time to better understand how discovery tools affect entire library collections. North Carolina State University Libraries released a final report about their usability study of Summon.31 The results of these usability studies were similar to other studies of discovery tools: users were satisfied with the ability to search the library catalog and article databases with a single search, but users had mixed results with known-item searching and confusion about narrowing facets and results ranking. Although several additional academic libraries have conducted usability studies of Encore, Summon, and EBSCO Discovery Service, the results have not yet been published.32

Only one usability study of EBSCO Discovery Service was found. In a study with six participants, Williams and Foster found users were satisfied and able to adapt to the new system quickly but did not take full advantage of the rich feature set.33

Combined with the rapid changes in these tools, the literature illustrates a current need for more usability studies related to discovery tools. The necessary focus on specific software implementations and different study designs make it difficult to identify common themes. Additional usability studies will offer greater breadth and depth to the current dialogue about discovery tools. This article will help fill the gap by presenting results from a usability study of EBSCO Discovery Service. Publishing such usability results of discovery tools will inform institutional decisions, improve user experiences, and advance the tools' content, features, and interface design. In addition, libraries will be able to more thoroughly modernize library catalogs to meet users' changing needs and expectations as well as keep pace with the evolution of the web.

METHOD

James Madison University Libraries' usability lab features one workstation with two pieces of usability software: Techsmith's Morae (version 3) (http://www.techsmith.com/morae.asp), which records screen captures of participant actions during the usability studies, and the Usability Testing Environment (UTE) (version 3) (http://www.utetool.com), which presents participants with tasks in a web-browser environment. The UTE also presents end-of-task questions to measure time on task and task success.

The study of EDS, conducted in October 2010, was covered by an institutional review board-approved protocol. Participants were recruited for the study through a bulk email sent to all students and faculty. Interested respondents were randomly selected to include a variety of grade levels and majors for students and years of service and disciplines taught for faculty members.

The study included ten participants with ranging levels of experience: two freshman, two sophomores, two juniors, one senior, one graduate student, and two faculty members. Three of the participants were from the school of business, one from education, two from the arts and humanities, and two from the sciences. The remaining two participants had dual majors in the humanities and the sciences. A usability rule of thumb is that at least five users will reveal more than 75 percent of usability issues.34 Because the goal was to observe a wide range of user behaviors and usability issues, and to gather data about satisfaction from a variety of perspectives, this study used two users of each grade level plus two faculty participants (for a total of ten) to provide as much heterogeneity as possible.

Student participants were presented with ten pre-study questions, and faculty participants were asked nine pre-study questions (see appendix A). The pre-study questions were intended to gather information about participants' background, including their time at JMU, their academic discipline, and their experience with the library website, the EBSCOhost interface, the library catalog, and library instruction. Since participants were anonymous, we hoped their answers would help us interpret unusual comments or findings. Pre-test results were not used to form comparison groups (e.g., freshmen versus senior) because these groups would not be representative of their larger populations. These questions were followed by a practice task to help familiarize participants with the testing software.

The study consisted of nine tasks designed to showcase usability issues, show the researchers how users behaved in the system, and measure user satisfaction. Appendix B lists the tasks and what they were intended to measure. In designing the test, determining success on some tasks seemed very objective (find a video about a given topic) while others appeared to be more subjective (those involving relevance judgments). For this reason, we asked participants to provide satisfaction information on some tasks and not others. In retrospect, for consistency of interpretation, we probably should have asked participants to rate or comment on every task. All of the tasks were presented in the same order. Tasks were completed either by clicking "Answer" and answering a question (multiple choice or typed response), or by clicking "Finished" after navigating to a particular webpage. Participants also had the option to skip the task they were working on and move to the next task. Allowing participants to skip a task helps differentiate between genuinely incorrect answers and incorrect answers due to participant frustration or guessing. A time limit of 5 minutes was set for tasks 1-7, while tasks 8 and 9 were given time limits of 8 minutes, after which the participant was timed out. Time limits were used to ensure participants were able to complete all tasks within the agreed-upon session. Average time on task across all tasks was 1 minute, 35 seconds.

After the study was completed, participants were presented with the System Usability Scale (SUS), a ten-item scale using statements of subjective assessment and covering a variety of aspects of system usability.35 SUS scores, which provide a numerical score out of 100, are affected by the complexity of both the system and the tasks users may have performed before taking the SUS. The SUS was followed by a post-test consisting of six open-ended questions, plus one additional question for faculty participants, intended to gather more qualitative feedback about user satisfaction with the system (see appendix A).

A technical glitch with the UTE software affected the study in two ways. First, on seven of the ninety tasks, the UTE failed to enforce the five-minute maximum time limit, and participants exceeding a task's time limit were allowed to continue the task until they completed or skipped the task. One participant exceeded the time limit on task 1 while three of these errors occurred during both tasks 8 and 9. This problem potentially limits the ability to compare the average time on task across tasks; however, since this study used time on task in a descriptive rather than comparative way, the impact on interpreting results is minimal. The seven instances in which the glitch occurred were included in the average time on task data found in figure 3 because the times were not extreme and the time limit had been imposed mostly to be sure participants had time to complete all the tasks. A second problem with the UTE was that it randomly and prematurely aborted some users' tasks; when this happened, participants were informed that their time had run out and were then moved on to the next task. This problem is more serious because it is unknown how much more time or effort the participant would have spent on the task or whether they would have been more successful. Because of this, the results below specify how many participants were affected for each task. Although this was unfortunate, the results of the participants who did not experience this problem still provide useful cases of user behavior, especially because this study does not attempt to generalize observed behavior or usability issues to the larger population. Although a participant mentioned a few technical glitches during testing to the facilitator, the extent of software errors was not discovered until after the tests were complete (and the semester was over) because the facilitator did not directly observe participants during sessions.

RESULTS

The participants were asked several pre-test questions to learn about their research habits. All but one participant indicated they used the library website no more than six times per month (see figure 2). Common tasks this study's student participants said they performed on the website were searching for books and articles, searching for music scores, "research using databases," and checking library hours. The two faculty participants mentioned book and database searches, electronic journal access, and interlibrary loan. Participants were shown the Quick Search widget and were asked "how much of the library's resources do you think the Quick Search will search?" Seven participants said "most"; only one person, a faculty member, said it would search "all" the library's resources.

When shown screenshots of the library catalog and an EBSCOhost database, seven participants were sure they had used LEO Library Catalog, and three were not sure. Three indicated that they had used an EBSCO database before, five had not, and two were not sure. Participants were also asked how often they had used library resources for assignments in their major field of study; four said "often," two said "sometimes," one "rarely/never," and one "very often." Students were also asked "has a librarian spoken to a class you've attended about library research?" and two said yes, five said no, and one was not sure.

A "practice task" was administered to ensure participants were comfortable with the workstation and software: "Use Quick Search to search a topic relating to your major/discipline or another topic of interest to you. If you were writing a paper on this topic how satisfied would you be with these results?" No one selected "no opinion" or very unsatisfied"; Sixty percent were "very satisfied" or "satisfied" with their results; forty percent were "somewhat unsatisfied."

The first task ("What was the last thing you searched for when doing a research assignment for class? Use Quick Search to re-search for this.") started participants on the library homepage. Participants were then asked to "Tell us how this compared to your previous experience" using a text box. The average time on task was almost 2 minutes; however one faculty participant took more than 12 minutes on this task; if his or her time was removed, the time on task average was 1 minute, 23 seconds. Figure 5 shows the participants' search terms and their comments.

The second task started on the library homepage and asked participants to find a video related to early childhood cognitive development. This task was chosen because JMU Libraries have significant video collections and because the research team hypothesized users might have trouble because there was no explicit way to limit to videos at the time. The average time on this task was two minutes, with one person experiencing an arbitrary time out by the software. Participants were judged to be successful on this task by the researchers if they found any video related to the topic. All participants were successful on this task, but four entered, then left the discovery tool interface to complete the task. Five participants looked for a video search option in the drop-down menu, and of these, three immediately used something other than Quick Search when they saw that there was no video search option. Of those who tried Quick Search, six opened the source type facet in EDS search results and four selected a source type limit, but only two selected a source type that led directly to success ("non-print resources").

Task 3 started participants in EDS (see figure 6) and asked them to search on speech pathology, find a way to limit search results to audiology, and limit their search results to peer-reviewed sources. Participants spent an average of 1 minute, 40 seconds on this task, with five participants being artificially timed out by the software. Participants' success on this task was determined by the researchers' examination of the number of subtasks they completed. The three subtasks consisted of successfully searching for the given topic (speech language pathology) limiting the search results to audiology, and further limiting the results to peer reviewed sources. Four participants were able to complete all three subtasks, including two who were timed out. (The times for those who were timed out were not included in time on task averages, but they were given credit for success.) Five completed just two of the subtasks, failing to limit to peerreviewed; one of these because of a timeout. It was unclear why the remaining participants did not attempt to alter the search results to "peer reviewed." Looking at the performed actions, six of the ten typed "AND audiology" into search keywords to narrow the search results, while one found and used "audiology" in the Subject facet on the search results page. Six participants found and used the "Scholarly (Peer Reviewed) Journals" checkbox limiter.

Beginning with the results they had from task 3, task 4 asked participants to find more recent sources and to select the most recent source available. Task success was measured by correct completion of two subtasks: limiting the search results to the last five years and finding the most recent source. The average time on task was 1 minute, 14 seconds, with three artificial timeouts. Of those who did not time out, all seven were able to limit their sources to be more recent in some way, but only three were able to select the most recent source. In addition to this being a common research task, the team was interested to see how users accomplished this task. Three typed in the limiter in the left-hand column, two typed in the limiter on the advanced search screen, and two used the date slider. Two participants used the "sort" drop-down menu to change the sort order to "Date Descending," which helped them complete this task. Other participants changed the dates, and then selected the first result, which was not the most recent.

Task 5, which started within EDS, asked participants to find a way to ask a JMU librarian for help. The success of this task was measured by whether they reached the correct URL for the Ask-a-Librarian page; eight of the ten participants were successful. This task took an average of only 31 seconds to complete, and eight of the ten used the Ask-a-Librarian link at the top of the page. Of the two unsuccessful participants, one was timed out, while another clicked "search modes" for no apparent reason, then clicked back and decided to finish the task.

Task 6 started in the EDS interface and asked participants to locate the journal Yachting and Boating World and select the correct coverage dates and online status from a list of four options; participants were deemed successful at two subtasks if they selected the correct option and successful at one subtask if they chose an option that was partially correct. Participants took an average of two minutes on this task; only five answered correctly. During this task, three participants used the EBSCO search option "SO Journal Title/Source," four used quotation marks, and four searched or re-searched with the "Title" drop-down menu option. Three chose the correct dates of coverage, but were unable to correctly identify the online availability. It is important to note that only searching and locating the journal title were accomplished with the discovery tool; to see dates of coverage and online availability, users clicked JMU's link resolver button, and the resulting screen was served from Serials Solutions' Article Linker product. Although some users spent more time than perhaps was necessary using the EDS search options to locate the journal, the real barriers to this task were encountered when trying to interpret the Serials Solutions screen.

Task 7, where participants started in EDS, was designed to determine whether users could navigate to a research database outside of EDS. Users were asked to look up the sculpture Genius of Mirth and were told the library database Camio would be the best place to search. They were instructed to "locate this database and find the sculpture." The researcher observed the recordings to determine success on this task, which was defined as using Camio to find the sculpture. Participants took an average of 1 minute, 32 seconds on this task; seven were observed to complete the task successfully, while three chose to skip the task. To accomplish this task, seven participants used the JMU Research Databases link in the header navigation at some point, but only four began the task by doing this. Six participants began by searching within EDS.

The final two tasks started on the library homepage and were a pair: participants were asked to find two books and two recent, peer-reviewed articles (from the last five years) on rheumatoid arthritis. Task 8 asked them to use the library's EDS widget, Quick Search, to accomplish this, and task 9 asked them to accomplish the same task without using Quick Search. When they found sources, they were asked to enter the four relevant titles in a text-entry box. The average time spent on these tasks was similar: about four minutes per task. Comparing these tasks was somewhat confusing because some participants did not follow instructions. User success was determined by the researchers' observation of how many of the four subtasks the user was able to complete successfully: find two books, find two articles, limit to peer reviewed, and select articles from last five years (with or without using a limiter); figure 4 shows their success.

Looking at the seven users who used Quick Search on the Quick Search tasks, six limited to "Scholarly (Peer Reviewed) Journals"; six limited to the last five years; and seven narrowed results using the source type facet. The average number of subtasks completed on task eight was 3.14 out of 4. Looking at the seven users who followed instructions and did not use Quick Search on task 9, all began with the library catalog and tried to locate articles within the library catalog. The average number of subtasks completed on task 9 was 2.29 out of 4. Some users tried to locate articles by setting the catalog's material type drop-down menu to "Periodicals" and others used the catalog's "Periodical" tab, which performed a title keyword search of the e-journal portal. For task 9, only two users eventually chose a research database to find articles. User behavior can only be compared for the six users (all students) who followed instructions on both tasks; a summary is provided in figure 4.

After completing all nine tasks, participants were presented with the System Usability Scale. EDS scored 56 out of 100. Following the SUS, participants were asked a series of post-test questions. Only one of the faculty members chose to answer the post-test questions. When asked how they would use Quick Search, all eight students explicitly mentioned class assignments, and the participating faculty member replied "to search for books." Two students mentioned books specifically, while the rest used the more generic term "sources" to describe items for which they would search. When asked "when would you not use this search tool?" the faculty member said "I would just have to get used to using it. I mainly go to [the library catalog] and then research databases." Responses from the six students who answered this question were vague and hard to categorize:

* "Not really sure for more general question/learning"

* "When just browsing"

* "For quick answers"

* "If I could look up the information on the internet"

* "When the material I need is broad"

* "Basic searching when you do not need to say where you got the info from"

When asked for the advantages of Quick Search, four specifically mentioned the ability to narrow results, three respondents mentioned "speed," three mentioned ease of use, and three mentioned relevance in some way (e.g., "it does a pretty good job associating keywords with sources"). Two mentioned the broad coverage and one compared it to Google, "which is what students are looking for." When asked to list disadvantages, the faculty member mentioned he/she was not sure what part of the library home page was actually "Quick Search," and was not sure how to get to his/her library account. Three students talked about Quick Search being "overwhelming" or "confusing" because of the many features, although one of these also stated, "like anything you need to learn in order to use it efficiently." One student mentioned the lack of an audio recording limit and another said "when the search results come up it is hard to tell if they are usable results."

Knowing that Quick Search may not always provide the best results, the research team also asked users what they would do if they were unable to find an item using Quick Search. A faculty participant said he or she would log into the library catalog and start from there. Five students mentioned consulting a library staff member in some fashion. Three mentioned moving on from library resources, although not necessarily as their first step. One said "find out more information on it to help narrow down my search." Only one student mentioned the library catalog or any other specific library resource.

When participants were asked if "Quick Search" was an appropriate name, seven agreed that it was. Of those who did not agree, one participant's comment was "not really, though I don't think it matters." And another's was "I think it represents the idea of the search, but not the action. It could be quicker." The only alternative name suggestion was "Search Tool."

Web Traffic Analysis

Web traffic through Quick Search and in EDS provides additional context for this study's results. During August-December 2010, Quick Search was searched 81,841 times from the library homepage. This is an increase from traffic into the previous widget in this location that searched the catalog, which received 41,740 searches during the same period in 2009. Even adjusting for an approximately 22 percent increase in website traffic from 2009 to 2010, this is an increase of 75 percent. Interestingly, the traffic to the most popular link on the library homepage, Research Databases, went from 55,891 in 2009 to 30,616 in 2010, a decrease of 55 percent when adjusting for the change in website traffic.

During fall 2010, 28 percent of Quick Search searches from the homepage were executed using at least one drop-down menu. Twelve percent changed Quick Search's first drop-down menu to something other than the keyword default, with "title" being the most popular option (7 percent of searches) followed by author (4 percent of searches). Twenty percent of users changed the second drop-down option; "Just Articles" and "Just Books" were the most popular options, garnering 7 percent and 6 percent of searches, respectively, followed by "Just Scholarly Articles," which accounted for 4 percent of searches.

Looking at EBSCO's statistical reports for JMU's implementation of EDS, there were 85,835 sessions and approximately 195,400 searches during August-December 2010. This means about 95 percent of EDS sessions were launched using Quick Search from the homepage. There were an average of 2.3 searches per session, which is comparable to past behavior in JMU's other EBSCOhost databases.

DISCUSSION

The goal of this study was to gather initial data about user behavior, usability issues, and user satisfaction with discovery tools. The task design and technical limitations of the study mean that comparing time on task between participants or tasks would not be particularly illuminating; and, while the success rates on tasks are interesting, they are not generalizable to the larger JMU population. Instead, this study provided observations of user behavior that librarians can use to improve services, it suggested some "quick fixes" to usability issues, and it pointed to several research questions. When possible, these observations are supplemented by comparisons between this study and the only other published usability study of EDS.36

This study confirmed a previous finding of user studies of federated search software and discovery tools: students have trouble determining what is searched by various systems.37 On the tasks in which they were asked to not use Quick Search to find articles, participants tried to search for articles in the library catalog. Although all but one of this study's participants correctly answered that Quick Search did not search "all" library resources, seven thought it searched "most." Both "most" or "some" would be considered correct; however, it is interesting that answering this question more specifically is challenging even for librarians. Many journals in subject article indexes and abstracts are included in the EDS Foundation Index; furthermore, JMU's implementation of EDS includes all of JMU's EBSCO subscription resources as well, making it impractical to assemble a master list of indexed titles. Of course, there are numerous online resources with contents which may never be included in a discovery tool, such as political voting records, ethnographic files, and financial data. Users often have access to these resources through their library. However, if they do not know the library has a database of financial data, they will certainly not consider this content in their response to a question of how many of the library resources are included in the discovery tool. As discovery tools begin to fulfill users' expectations for a "single search," libraries will need to share best practices for showcasing valuable, useful collections that fall outside the discovery tool's scope or abilities. This is especially critical when reviewing the 72 percent increase in homepage traffic to the homepage search widget compared with the 55 percent decrease in homepage traffic to the research databases page. It is important to note these trends do not mean the library's other research databases have fallen in usage by 55 percent. Though there was not a comprehensive examination of usage statistics, spot-checking suggested EBSCO and non-EBSCO subject databases had both increases and decreases in usage from previous years.

Another issue libraries should consider, especially when preparing for instruction classes, is that users do not seem to understand which information needs are suited to a discovery tool versus the catalog or subject-specific databases. Several tasks provided additional information about users' mental models of the tool, which may help libraries make better decisions about navigation customizations in discovery tool interfaces and on library websites. Task 7 was designed to discover whether users could find their way to a database outside of EDS if they knew they needed to use a specific database. Six participants, including one of the faculty members, began by searching EDS for the name of the sculpture and/or the database name. On task 1, a graduate student who searched on "Death of a Salesman" and was asked to comment on how Quick Search results compared to his or her previous experience, said, "I would still have liked to see more critical sources on the play but I could probably have found more results of that nature with a better search term, such as 'death of a salesman criticism.'" While true, most librarians would suggest using a literary criticism database, which would target this information need.

Librarians may have differing opinions regarding the best research starting point, but their rationale would be much different than that of the students in this study. This study's participants said they would use Quick Search/EDS when they were doing class work or research, but would not use it for general inquiries. If librarians were to list which user information needs are best met by a discovery tool versus a subject-specific database, the types of information needs listed would be much more numerous and diverse, regardless of differences over how to classify them.

In addition to helping users choose between a discovery tool or a subject-specific database, libraries will need to conceptualize how users will move in and out of the discovery tool to other library resources, services, and user accounts. While users had no trouble finding the Ask-a-Librarian link in the header, it might have been more informative if users started from a search-results page to see if they would find the right-hand column's Ask-a-Librarian link or links to library subject guides and database lists. Discovery tools vary in their abilities to connect users with their online library accounts and are changing quickly in this area.

This study also provided some interesting observations about discovery tool interfaces. The default setting for EBSCO Discovery Service is a single search box. However, this study suggests that while users desire a single search, they are willing to use multiple interface options. This was supported by log analysis of the library's locally developed entry widget, Quick Search, in which 28 percent of searches included the use of a drop-down menu. On the first usability task, users left Quick Search's options set to the default. On other tasks, participants frequently used the drop-down menus and limiters in both Quick Search and EDS. For example, on task 2, which asked them to look for videos, five users looked in the Quick Search format drop-down menu. On the same task within EDS, six users attempted to use the source type facet. Use of limiters was similarly observed by Williams and Foster in their EDS usability study.38

One EDS interface option that was not obvious to participants was the link to change the sort order. When asked to find the most recent article, only two participants changed the sort option. Most others used the date input boxes to limit their search, then selected the first result even thought it was not the most recent one. It is unclear whether the participant assumed the first result was the most recent or whether they could not figure out how to display the most recent sources.

Finding a journal title from library homepages has long been a difficult task,39 and this study provided no exception, even with the addition of a discovery tool. It is important to note that the standard EDS implementation would include a "Publications" or "Journals A-Z" link in the header; in EDS, libraries can customize the text of this link. JMU did not have this type of link enabled in our test, since the hope was that users could find journal titles within the EDS results. However, neither EDS nor the Quick Search widget's search interfaces offered a way to limit the search to a journal title at the time of this study. During the usability test, four participants changed the field search drop-down menu to "Title" in EDS, and three participants changed the EDS field search drop-down menu to "SO Journal Title/Source," which limits the search to articles within that journal title. While both of these ideas were good, neither one resulted in a precise results set in EDS for this task unless the user also limited to "JMU Catalog Only," a nonintuitive option. Since the test, JMU has added a "Journal Titles" option to Quick Search that launches the user's search into the journal A-Z list (provided by Serials Solutions). In two months after the change (February and March 2011), only 391 searches were performed with this option. This was less than 1 percent of all searches, indicating that while it may be an important task, it is not a popular one.

Like many libraries with discovery tools, JMU added federated search capabilities to EDS using EBSCOhost Integrated Search software in an attempt to draw some traffic to databases not included in EDS (or not subscribed to through EBSCO by JMU), such as MLA International Bibliography, Scopus, and Credo Reference. Links to these databases appeared in the upper-right-hand column of EDS during the usability study (see figure 6.) Usage data from EBSCO showed that less than 1 percent of all JMU's EDS sessions for fall 2010 included any interaction with this area. Likewise, Williams and Foster observed their participants did not use their federated search until explicitly asked to do so.40 Perhaps users faced with discovery tool results simply have no motivation to click on additional database results. Since the usability test, JMU has replaced the right-hand column with static links to Ask-a-Librarian, subject guides, and research database lists.

Readers may wonder why one of the most common tasks, finding a specific book title, was not included in this usability study; this was because JMU Libraries posed this task in a concurrent homepage usability study. On that study, twenty of the twenty-five participants used Quick Search to find the title "Pigs in Heaven" and choose the correct call number. Eleven of the twenty used the Quick Search drop-down menu to choose a title search option, further confirming users' willingness to limit up-front. The average time on this task was just under a minute, and all participants completed this task successfully, so this task was not repeated in the EDS usability test. Other studies have reported trouble with this type of task;41 much could depend on the item chosen as well as the tool's relevance ranking.

User satisfaction with EDS can be summarized from the open-ended post-study questions, from the responses to task 1 (figure 5), and the SUS scale. Answers to the post-study questions indicated participants liked the ability to narrow results, the speed and ease of use, and relevance of the system. A few participants did describe the system as being "overwhelming" or "confusing" because of the many features, which was also supported by the SUS scores. JMU has been using the SUS to understand the relative usability of library systems. The SUS offers a benchmark for system improvement; for example, EBSCO Discovery Service received an SUS of only 37 in spring 2010 (N = 7) but a 56 on this study in fall 2010 (N = 10). This suggests the interface has become more usable. In 2009, JMU Libraries also used the SUS to test the library catalog's classic interface as well as a VuFind interface to the library catalog, which received scores of 68 (N = 15) and 80 (N = 14), respectively. The differences between the catalog scores and EDS indicate an important distinction between usability and usefulness, with the latter concept encompassing a system's content and capabilities. The library catalog is, perhaps, a more straightforward tool than a discovery tool and attempts to provide access to a smaller set of information. It has none of the complexity involved in finding article-level or book chapter information. All else being equal, simpler tools will be more usable. In an experimental study, Tsakonas and Paptheodorou found that while users did not distinguish between the concepts of usability and usefulness, they prefer attributes composing a useful system in contrast to those supporting usability.42 Discovery tools, which support more tasks, must make compromises in usability that simpler systems can avoid. In their study of EDS, Williams and Foster also found overall user satisfaction with EDS. Their participants made positive comments about the interface as well as the usefulness and relevance of the results.43

JMU passed on several suggestions to EBSCO related to EDS based on the test results. EBSCO subsequently added "Audio" and "Video" to the source types, which enabled JMU to add a "Just Videos at JMU" option to Quick Search. While it is confusing that "Audio" and "Video" source types currently behave differently than the others in EDS, in that they limit to JMU's catalog as well as to the source type, this behavior produces what most local users expect. A previous usability study of WorldCat Local showed users have trouble discriminating between source types in results lists, so the source types facet is important.44 Another piece of feedback provided to EBSCO was that on the task where users needed to choose the most recent result, only two of our participants sorted by date descending. Perhaps the textual appearance of the sort option (instead of a drop-down menu) was not obvious to participants (see figure 6); however, Williams and Foster did not observe this to be an issue in their study.45

FUTURE RESEARCH

The findings of this study suggest many avenues for future research. Libraries will need to revisit the scope of their catalogs and other systems to keep up with users' mental models and information needs. Catalogs and subject-specific databases still perform some tasks much better than discovery tools, but libraries will need to investigate how to situate the discovery tool and specialized tools within their web presence in a way that will make sense to users. When should a user be directed to the catalog versus a discovery tool? What items should libraries continue to include in their catalogs? What role do institutional repositories play in the suite of library tools, and how does the discovery tool connect to them (or include them?) How do library websites begin to make sense of the current state of library search systems? Above all, are users able to find the best resources for their research needs? Although research on searchers' mental models has been extensive,46 librarians' mental models have not been studied as such. Yet placing the discovery tool among the library's suite of services will involve compromises between these two models.

Another area needing research is how to instruct users to work with the large numbers of results returned by discovery tools. In subject-specific databases, librarians often help users measure the success of their strategy-or even their topic-by the number of results returned: in Criminal Justice Abstracts, 5,000 results means a topic is too broad or the search strategy needs refinement. In a discovery tool, a result set this large will likely have some good results on the first couple of pages if sorted by relevance; however, users will still need to know how to grow or reduce their results sets. Participants in this study showed a willingness to use limiters and other interface features, but not always the most helpful ones. When asked to narrow a broad subject on task 3 of this study, only one participant chose to use the "Subject" facet even when the subtopic, audiology, was clearly available. Most added search terms. It will be important for future studies to investigate the best way for users to narrow large results set in a discovery tool.

This study also suggested possible areas of investigation for future user studies. One interesting finding related to this study's users' information contexts was that when users were asked to search on their last research topic, it did not always match up with their major: a voice performance student searched on "current issues in Russia," and the hospitality major searched on "aphasia." To what extent does a discovery tool help or hinder students who are searching outside their major area of study? One of JMU's reference librarians noted that while he would usually teach a student majoring in a subject how to use that subject's specific indexes, as opposed to a discovery tool, a student outside the major might not need to learn the subject-specific indexes for that subject and could be well served by the discovery tool. Future studies could also investigate the usage and usability of discovery tool features in order to continue informing library customizations and advice to vendors. For example, this study did not have a task related to logging into a patron account or requesting items, but that would be good to investigate in a follow-up study. Another area ripe for further investigation is discovery tool limiters. This study's participants frequently attempted to use limiters, but didn't always choose the correct ones for the task. What are the ideal design choices for making limiters intuitive? This study found almost no use of the embedded federated search add-on: is this true at other institutions? Finally, this study and others reveal difficulty in distinguishing source types. Development and testing of interface enhancements to support this ability would be helpful to many libraries' systems.

CONCLUSION

This usability test of a discovery tool at James Madison University did not reveal as many interface-specific findings as it did questions about the role of discovery tools in libraries. Users were generally able to navigate through the Quick Search and EDS interfaces and complete tasks successfully. Tasks that are challenging in other interfaces, such as locating journal articles and discriminating between source types, continued to be challenging in a discovery tool interface.

This usability test suggested that while some interface features were heavily used, such as drop-down limits and facets, other features were not used, such as federated search results. As discovery tools continue to grow and evolve, libraries should continue to conduct usability tests, both to find usability issues and to understand user behavior and satisfaction. Although discovery tools challenge libraries to think not only about access but also about the best research pathways for users, they provide users with a search that more closely matches their expectations.

ACKNOWLEDGEMENT

The authors would like to thank Patrick Ragland for his editorial assistance in preparing this manuscript.

References

REFERENCES

1. Emily Alling and Rachael Naismith, "Protocol Analysis of a Federated Search Tool: Designing for Users," Internet Reference Services Quarterly 12, no. 1 (2007): 195, http://scholarworks.umass.edu/librarian_pubs/1/ (accessed Jan. 11, 2012); Frank Cervone, "What We've Learned From Doing Usability Testing on OpenURL Resolvers and Federated Search Engines," Computers in Libraries 25, no. 9 (2005): 10 ; Sara Randall, "Federated Searching and Usability Testing: Building the Perfect Beast," Serials Review 32, no. 3 (2006): 181-82, doi:10.1016/j.serrev.2006.06.003; Ed Tallent, "Metasearching in Boston College Libraries-A Case Study of User Reactions," New Library World 105, no. 1 (2004): 69-75, DOI: 10.1108/03074800410515282.

2. S. C. Williams and A. K. Foster, "Promise Fulfilled? An EBSCO Discovery Service Usability Study," Journal of Web Librarianship 5, no. 3 (2011), http://www.tandfonline.com/doi/pdf/10.1080/19322909.2011.597590 (accessed Jan. 11, 2012).

3. Janet K. Chisman, Karen R. Diller, and Sharon L. Walbridge, "Usability Testing: A Case Study," College & Research Libraries 60, no. 6 (November 1999): 552-69, http://crl.acrl.org/content/60/6/552.short (accessed Jan. 11, 2012); Frances C. Johnson and Jenny Craven, "Beyond Usability: The Study of Functionality of the 2.0 Online Catalogue," New Review of Academic Librarianship 16, no. 2 (2010): 228-50, DOI: 10.1108/00012531011015217 (accessed Jan, 11, 2012); Jennifer E. Knievel, Jina Choi Wakimoto, and Sara Holladay, "Does Interface Design Influence Catalog Use? A Case Study," College & Research Libraries 70, no. 5 (September 2009): 446-58, http://crl.acrl.org/content/70/5/446.short (accessed Jan. 11, 2012); Jia Mi and Cathy Weng, "Revitalizing the Library OPAC: Interface, Searching, and Display Challenges," Information Technology & Libraries 27, no. 1 (March 2008): 5-22, http://0-www.ala.org.sapl.sat.lib.tx.us/ala/mgrps/divs/lita/publications/ital/27/1/mi.pdf (accessed Jan. 11, 2012).

4. Karen Calhoun, "The Changing Nature of the Catalog and its Integration with Other Discovery Tools," http://www.loc.gov/catdir/calhoun-report-final.pdf (accessed Mar. 11, 2011).

5. Dee Ann Allison, "Information Portals: The Next Generation Catalog," Journal of Web Librarianship 4, no. 1 (2010): 375-89, http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=1240&context=libraryscience (accessed January 11, 2012); Marshall Breeding, "The State of the Art in Library Discovery," Computers in Libraries 30, no. 1 (2010): 31-34; C. P Diedrichs, "Discovery and Delivery: Making it Work for Users . . . Taking the Sting out of Serials!" (lecture, North American Serials Interest Group, Inc. 23rd Annual Conference, Phoenix, Arizona, June 5-8, 2008), DOI: 10.1080/03615260802679127; Ian Hargraves, "Controversies of Information Discovery," Knowledge, Technology & Policy 20, no. 2 (Summer 2007): 83, http://www.springerlink.com/content/au20jr6226252272/fulltext.html (accessed Jan. 11, 2012); Jane Hutton, "Academic Libraries as Digital Gateways: Linking Students to the Burgeoning Wealth of Open Online Collections," Journal of Library Administration 48, no. 3 (2008): 495-507, DOI: 10.1080/01930820802289615; OCLC, "Online Catalogs: What Users and Librarians Want: An OCLC Report," http://www.oclc.org/reports/onlinecatalogs/default.htm (accessed Mar. 11 2011).

6. C. J. Belliston, Jared L. Howland, and Brian C. Roberts, "Undergraduate Use of Federated Searching: A Survey of Preferences and Perceptions of Value-Added Functionality," College & Research Libraries 68, no. 6 (November 2007): 472-86, http://crl.acrl.org/content/68/6/472.full.pdf+html (accessed Jan. 11, 2012); Judith Z. Emde, Sara E. Morris, and Monica Claassen-Wilson, "Testing an Academic Library Website for Usability with Faculty and Graduate students," Evidence Based Library & Information Practice 4, no. 4 (2009): 24-36, http://kuscholarworks.ku.edu/dspace/bitstream/1808/5887/1/emdee_morris_CW.pdf (accessed Jan. 11,2012); Karla Saari Kitalong, Athena Hoeppner, and Meg Scharf, "Making Sense of an Academic Library Web Site: Toward a More Usable Interface for University Researchers," Journal of Web Librarianship 2, no. 2/3 (2008): 177-204, http://www.tandfonline.com/doi/abs/10.1080/19322900802205742 (accessed Jan. 11, 2012); Ed Tallent, "Metasearching in Boston College Libraries-A Case Study of User Reactions," New Library World 105, no. 1 (2004): 69-75, DOI: 10.1108/03074800410515282; Rong Tang, Ingrid Hsieh-Yee, and Shanyun Zhang, "User Perceptions of MetaLib Combined Search: An Investigation of How Users Make Sense of Federated Searching," Internet Reference Services Quarterly 12, no. 1 (2007): 211-36, http://www.tandfonline.com/doi/abs/10.1300/J136v12n01_11 (accessed Jan. 11, 2012).

7. Jody Condit Fagan, "Usability Studies of Faceted Browsing: A Literature Review," Information Technology & Libraries 29, no. 2 (2010): 58-66, http://web2.ala.org/ala/mgrps/divs/lita/publications/ital/29/2/fagan.pdf (accessed Jan. 11, 2012).

8. Birong Ho, Keith Kelley, and Scott Garrison, "Implementing VuFind as an Alternative to Voyager's Web Voyage Interface: One Library's Experience," Library Hi Tech 27, no. 1 (2009): 8292, DOI: 10.1108/07378830910942946 (accessed Jan. 11, 2012).

9. Tamar Sadeh, "User Experience in the Library: A Case Study," New Library World 109, no. 1 (2008): 7-24, DOI: 10.1108/03074800810845976 (accessed Jan. 11, 2012).

10. Tod A. Olson, "Utility of a Faceted Catalog for Scholarly Research," Library Hi Tech 25, no. 4 (2007): 550-61, DOI: 10.1108/07378830710840509 (accessed Jan. 11, 2012).

11. Allison, "Information Portals," 375-89.

12. Marshall Breeding, "Plotting a New Course for Metasearch," Computers in Libraries 25, no. 2 (2005): 27.

13. Ibid.

14. Dennis Brunning and George Machovec, "Interview About Summon with Jane Burke, Vice President of Serials Solutions," Charleston Advisor 11, no. 4 (2010): 60-62; Dennis Brunning and George Machovec, "An Interview with Sam Brooks and Michael Gorrell on the EBSCOhost Integrated Search and EBSCO Discovery Service," Charleston Advisor 11, no. 3 (2010): 62-65, http://www.ebscohost.com/uploads/discovery/pdfs/topicFile-121.pdf (accessed Jan. 11, 2012).

15. Ronda Rowe, "Web-Scale Discovery: A Review of Summon, EBSCO Discovery Service, and WorldCat Local," Charleston Advisor 12, no. 1 (2010): 5-10; K. Stevenson et al., "Next-Generation Library Catalogues: Reviews of Encore, Primo, Summon and Summa," SERIALS 22, no. 1 (2009): 68-78.

16. Jason Vaughan, "Chapter 7: Questions to Consider," Library Technology Reports 47, no. 1 (2011): 54; Paula L. Webb and Muriel D. Nero, "OPACs in the Clouds," Computers in Libraries 29, no. 9 (2009): 18.

17. Jason Vaughan, "Investigations into Library Web Scale Discovery Services," Articles (Libraries), paper 44 (2011), http://digitalcommons.library.unlv.edu/lib_articles/44.

18. Marshall Breeding, "The State of the Art in Library Discovery," 31-34; Sharon Q. Yang and Kurt Wagner, "Evaluating and Comparing Discovery Tools: How Close are We Towards Next Generation Catalog?" Library Hi Tech 28, no. 4 (2010): 690-709.

19. Allison, "Information Portals," 375-89.

20. Breeding, "The State of the Art in Library Discovery," 31-34.

21. Galina Letnikova, "Usability Testing of Academic Library Websites: A Selective Bibliography," Internet Reference Services Quarterly 8, no. 4 (2003): 53-68.

22. Jeffrey Rubin and Dana Chisnell, Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests, 2nd ed. (Indianapolis, IN: Wiley, 2008); Joseph S. Dumas and Janice Redish, A Practical Guide to Usability Testing, rev. ed. (Portland, OR: Intellect, 1999).

23. Nicole Campbell, ed., Usability Assessment of Library-Related Web Sites: Methods and Case Studies (Chicago: Library & Information Technology Association, 2001); Elaina Norlin and C. M. Winters, Usability Testing for Library Web Sites: A Hands-On Guide (Chicago: American Library Association, 2002).

24. Jennifer L. Ward, Steve Shadle, and Pam Mofield, "User Experience, Feedback, and Testing," Library Technology Reports 44, no. 6 (2008): 17.

25. Ibid.

26. Michael Boock, Faye Chadwell, and Terry Reese, "WorldCat Local Task Force Report to LAMP," http://hdl.handle.net/1957/11167 (accessed Mar. 11 2011).

27. Bob Thomas and Stefanie Buck, "OCLC's WorldCat Local Versus III's WebPAC: Which Interface is Better at Supporting Common User Tasks?" Library Hi Tech 28, no. 4 (2010): 648-71.

28. OCLC, "Some Findings from WorldCat Local Usability Tests Prepared for ALA Annual," http://www.oclc.org/worldcatlocal/about/213941usf_some_findings_about_worldcat_local.pdf (accessed Mar. 11, 2011).

29. Ibid., 2.

30. Doug Way, "The Impact of Web-Scale Discovery on the Use of a Library Collection," Serials Review 36, no. 4 (2010): 21420.

31. North Carolina State University Libraries, "Final Summon User Research Report," http://www.lib.ncsu.edu/userstudies/studies/2010_summon/ (accessed Mar. 28, 2011).

32. Alesia McManus, "The Discovery Sandbox: Aleph and Encore Playing Together," http://www.nercomp.org/data/media/Discovery%20Sandbox%20McManus.pdf (accessed Mar. 28, 2011); PRWeb, "Deakin University in Australia Chooses EBSCO Discovery Service," http://www.prweb.com/releases/Deakin/ChoosesEDS/prweb8059318.htm (accessed Mar. 28, 2011); University of Manitoba, "Summon Usability: Partnering with the Vendor," http://prezi.com/icxawthckyhp/summon-usability-partnering-with-the-vendor (accessed Mar. 28, 2011).

33. Williams and Foster, "Promise Fulfilled?"

34. Jakob Nielsen, "Why You Only Need to Test with 5 Users," http://www.useit.com/alertbox/20000319.html (accessed Aug. 20, 2011).

35. John Brooke, "SUS: A 'Quick and Dirty' Usability Scale," in Usability Evaluation in Industry, ed. P. W. Jordanet al. (London: Taylor & Francis, 1996), http://www.usabilitynet.org/trump/documents/Suschapt.doc (accessed Apr. 6, 2011).

36. Williams and Foster, "Promise Fulfilled?"

37. Seikyung Jung et al., "LibraryFind: System Design and Usability Testing of Academic Metasearch System," Journal of the American Society for Information Science & Technology 59, no. 3 (2008): 375-89; Williams and Foster, "Promise Fulfilled?"; Laura Wrubel and Kari Schmidt, "Usability Testing of a Metasearch Interface: A Case Study," College & Research Libraries 68, no. 4 (2007): 292-311.

38. Williams and Foster, "Promise Fulfilled?"

39. Letnikova, "Usability Testing of Academic Library Websites," 53-68; Tom Ipri, Michael Yunkin, and Jeanne M. Brown, "Usability as a Method for Assessing Discovery," Information Technology & Libraries 28, no. 4 (2009): 181-86; Susan H. Mvungi, Karin de Jager, and Peter G. Underwood, "An Evaluation of the Information Architecture of the UCT Library Web Site," South African Journal of Library & Information Science 74, no. 2 (2008): 171-82.

40. Williams and Foster, "Promise Fulfilled?"

41. Ward et al., "User Experience, Feedback, and Testing," 17.

42. Giannis Tsakonas and Christos Papatheodorou, "Analysing and Evaluating Usefulness and Usability in Electronic Information Services," Journal of Information Science 32, no. 5 (2006): 400-419.

43. Williams and Foster, "Promise Fulfilled?"

44. Bob Thomas and Stefanie Buck, "OCLC's WorldCat Local Versus III's WebPAC: Which Interface is Better at Supporting Common User Tasks?" Library Hi Tech 28, no. 4 (2010): 648-71.

45. Williams and Foster, "Promise Fulfilled?"

46. Tracy Gabridge, Millicent Gaskell, and Amy Stout, "Information Seeking through Students' Eyes: The MIT Photo Diary Study," College & Research Libraries 69, no. 6 (2008): 510-22; Yan Zhang, "Undergraduate Students' Mental Models of the Web as an Information Retrieval System," Journal of the American Society for Information Science & Technology 59, no. 13 (2008): 2087-98; Brenda Reeb and Susan Gibbons, "Students, Librarians, and Subject Guides: Improving a Poor Rate of Return," Portal: Libraries and the Academy 4, no. 1 (2004): 123-30; Alexandra Dimitroff, "Mental Models Theory and Search Outcome in a Bibliographic Retrieval System," Library & Information Science Research 14, no. 2 (1992): 141-56.

Appendix

(ProQuest: Appendix omitted.)