Simplifying and Enhancing Access to Full-Text Articles with LibKey Discovery

Abstract

Content delivery mechanisms in web-scale discovery services can often fall short of patron expectations. The Texas Tech University Libraries integrated Third Irons LibKey Discovery with Ex Libris Primo in response to patron feedback that accessing full-text articles from Primo required too many clicks. Following implementation, researchers conducted user testing to assess the usability and usefulness of the integration. They also analyzed LibKey Discovery and link resolver usage data across the year following the launch to evaluate its impact on how patrons accessed full-text articles. Patrons responded positively overall to LibKey Discovery, perceiving it as more efficient and quicker to use. Statistics showed an increase in LibKey link usage and a corresponding decrease in link resolver clickthroughs. This article provides an overview of the implementation process, describes assessment methodologies and findings, and discusses implications for improving usability and increasing usage.

Headnote

ABSTRACT

Content delivery mechanisms in web-scale discovery services can often fall short of patron expectations. The Texas Tech University Libraries integrated Third Irons LibKey Discovery with Ex Libris Primo in response to patron feedback that accessing full-text articles from Primo required too many clicks. Following implementation, researchers conducted user testing to assess the usability and usefulness of the integration. They also analyzed LibKey Discovery and link resolver usage data across the year following the launch to evaluate its impact on how patrons accessed full-text articles. Patrons responded positively overall to LibKey Discovery, perceiving it as more efficient and quicker to use. Statistics showed an increase in LibKey link usage and a corresponding decrease in link resolver clickthroughs. This article provides an overview of the implementation process, describes assessment methodologies and findings, and discusses implications for improving usability and increasing usage.

INTRODUCTION

Academic libraries have widely adopted web-scale discovery services to facilitate discovery and delivery of the resources they provide to their patrons. Today's patrons are accustomed to simple search interfaces and one-click access to the information they need; and although web-scale discovery services were developed with user experience in mind, patron expectations and the realities of web-scale discovery service functionality are often at odds. Products such as Ex Libris Primo offer the simple, single-search box interface patrons are used to, but reliable, one-click access to full-text articles and other e-resources remains a persistent problem. User Experience (UX) and Systems personnel at the Texas Tech University Libraries (TTU Libraries) frequently heard from patrons that getting from the search results to the full text of an article required too many clicks, there were often too many linking options to choose from, sometimes the links didnt work or directed patrons to the wrong resource, and they didnt like having to figure out how to read or download articles on the provider's platform-all they wanted was the PDF. To address these pain points, librarians in the Systems and UX units explored options for simplifying and enhancing access to full-text articles and ultimately decided to implement Third Irons LibKey Discovery integration in the TTU Libraries Primo environment. This article will provide an overview of the implementation project, describe the implementation process, discuss the methods used to assess the efficacy and impact of the integration, and review the next steps for the project.

LITERATURE REVIEW

Library search interfaces have taken many forms in response to evolving patron expectations regarding discovery and delivery of resources. For decades patrons used card catalogs as their primary search "interface"; but as technology progressed during the late 1960s and early 1970s the card catalog evolved into the online public access catalog (OPAC), which functioned as a computer-based finding aid for a library's physical holdings.1 OPACs did not include electronic holdings, and the proliferation of full-text resources available online in the late 1990s necessitated the development of new discovery models and delivery mechanisms. Libraries initially provided lists of databases and ejournals with static links to database or journal websites in order to facilitate discovery and delivery of electronic collections, but the lack of a unified search interface meant that patrons had to perform separate searches on multiple platforms to find full-text articles relevant to their research. Federated search (or metasearch) tools allowed users to search multiple sources with one query and were popular from about 2000 to 2009; however, these products were not well-regarded by patrons due to their slow performance, shallow result sets, and unintuitive user interfaces.2

Discovery interfaces, also referred to as "next generation library catalogs," emerged in the mid-2000s in response to the need for an end-user interface that would provide both discovery and delivery of a library's physical and electronic holdings utilizing the simplified search processes and navigation conventions patrons had become accustomed to using in the general web environment Federated search tools and discovery interfaces initially provided title-level linking rather than article-level linking, and patrons had to search for an article again after being directed to the journal on the publisher or aggregator's website.4 In contrast, web-scale discovery services combine a discovery interface with a central index, providing a single search across the library's local collections and subscription content as well as article-level results and linking.5 Web-scale discovery services have been widely implemented by academic libraries in the interest of improving discovery as well as patrons' search experience; but as Tay points out, access and delivery are as equally important as discovery.6

Patrons typically don't want to just discover resources; they want to obtain the full text. Linking errors and usability issues are persistent problems that hinder patrons' ability to access full-text resources. As content became increasingly available from multiple sources and libraries began investing more in electronic formats, it became more difficult (if not impossible) for libraries to maintain reliable links to their licensed content. The OpenURL framework and link resolver software products were developed beginning in the late 1990s in response to the need for a context-sensitive linking mechanism that would resolve the "appropriate copy problem" by taking bibliographic metadata, a patron's institutional affiliation, and the library's holdings into account in order to construct links to content licensed by that patron's library.7 Stuart, Varnum, and Ahronheim point out that while OpenURL has become the glue that connects citations to full text, it is well recognized that link resolution is an imperfect science.8 Incorrect or incomplete metadata, inaccurate knowledge base holdings, and linking syntax problems are all common causes of OpenURL linking failures; however, usability issues also impact patrons' ability to access full-text articles.' Hanrath and Kottman's usability testing of Primo at the KU Libraries included tasks that required participants to locate and access a full-text resource, and they observed that the participants struggled with "using the link resolver page and navigating the resource platform."!° They also noted that several participants struggled to select a platform from the link resolver page when multiple services were available and that "the usability of the link resolver and the platforms accessed from there greatly impacted the success of the participants" in obtaining full-text resources.11 In their investigation of student perceptions of Primo, Hamlett and Georgas found that the student participants felt that Primo was easy to use, but "one student noted that they were confused by clicking on a title and not being led directly to the full text," and "7 out of 30 (23.3%) students indicated that they had difficulty finding or opening the full-text of articles."12 Walton, Childs, and Palumbo compared the user experience of Primo and EBSCO Discovery Service (EDS) and found that "the most prevalent difficulty that students encountered was understanding the concept of full text" and "many participants had difficulty distinguishing between the varying language prompts for access to the full text of resources."13 They also noted that "participants spent the most significant amount of time deciding between full-text options, often hovering over the various options for a period of time before deciding which one to use," indicating a need for consistent and improved labeling for full-text resources.14

Efforts to alleviate these problems have taken the form of direct links and one-click access to fulltext resources from discovery service search results. Breeding states that direct links are "often implemented in discovery services to provide more reliable access to electronic resources than through the OpenURL process, making use of internal or proprietary data beyond what would be available through OpenURL."15 EBSCOhost SmartLinks+, Ex Libris's Summon Index-Enhanced Direct Linking, and Ex Libris's Primo Quicklinks are some examples of direct linking methods offered by discovery service vendors; and Third Iron's LibKey Discovery is an example of third-party discovery service integration that enables direct linking to full-text articles. Simmons College enabled SmartLinks+ and adjusted link resolver menu display settings in their EDS instance after usability testing of EDS showed that students "preferred direct access to articles over linking to an article through the link resolver" and that they "expect direct access to full text articles from their results list.16 The University of Michigan Library enabled Index-Enhanced Direct Linking in their Summon instance in February 2012 in response to a high number of problem reports due to poor quality OpenURL resolution via the 360 Link link resolver.17 In early 2013, they tested linking via 360 Link's 1-Click feature and Summon's Index-Enhanced Direct Linking and found that "between Direct Linking and 1-Click about 93% to 94% of the time an attempt was made to lead users directly to the full text of the article without first going through the 360 Link menu;" and "Direct Linking led to full text more than 90% of the time while 1-Click led to full text from about 58% to about 67% of the time."18 Silton reported that Summon's Index-Enhanced Direct Linking's full-text delivery effectiveness ranged from 90.5% to 100% following its implementation at North Carolina A&T State University's Bluford Library in late 2011.19 Locascio and Rubel conducted a side-by-side comparison of Ex Libris's Primo Quicklinks and Third Iron's LibKey Discovery to analyze the frequency with which the direct linking tools' links displayed in brief display records; the linking success rate of each tool; and the number of clicks required to reach the full-text PDF from each link.20 They found that both direct linking tools had high rates of link success and required a similar average number of clicks to reach the full text, but LibKey Discovery links appeared in brief display records far more often than Quicklinks.

PROJECT OVERVIEW

Background

Ejournals represent a significant portion of the TTU Libraries' collections, so when patrons expressed dissatisfaction with Primo's linking to full-text journal articles, librarians in the Systems and UX units decided to take a closer look at the discovery-to-delivery workflow and explore enhancement options. The TTU Libraries utilize Ex Libris's Alma library management system (LMS) and Primo discovery service to provide resource discovery and delivery to patrons. In Primo, discovery is powered by data harvested from Alma and the Central Discovery Index (CDI),21 and delivery of e-resources is facilitated by the Alma link resolver and CDI Link in Record links.?? The TTU Libraries' Primo environment is configured in such a way that in order to navigate to full-text resources, users must click on a Brief Display record from their search results to open its Full Display page, then select one of the access options (link resolver service or CDI Link in Record link) available for the resource from the Full Display page's View It menu in order to navigate to the full text at the publisher or aggregator platform.

At the TTU Libraries, link resolver services are the primary access option available for scholarly journal articles. Per Ex Libris's product documentation, the link resolver workflow initiates when a patron clicks on a Primo search result or on a link resolver link available in other discovery tools such as Google Scholar, at which point an OpenURL request is sent to the link resolver. Upon receiving the OpenURL request, the link resolver extracts and parses the OpenURL metadata to build the Context Object, then uses the metadata stored in the Context Object to look for matching bibliographic records with active electronic holdings in Alma. When a match is found, the link resolver calculates relevant services based on availability and lists them in the View It menu in a Full Display page if the resource was discovered in Primo, or in the View It menu in a Services page if the patron clicked on a link resolver link for a resource discovered in another discovery tool. The link resolver generates the target URL that directs the patron to the resource once an available service is selected from a Full Display or Services page.23 In contrast, CDI Link in Record links utilize static URLs stored in CDI records and are primarily used for streaming videos, images, reference resources, or other resources that do not have standard identifiers.24

Regardless of linking method provided, accessing full-text articles from Primo requires a lot of clicks, especially from the perspective of users who are accustomed to the simplicity and one-click access to information provided by Google and Google Scholar. While itis possible to configure Primo to allow users to skip the Full Display page and link directly to resources from Brief Display records in search results, the TTU Libraries have opted to not enable that feature so that users can see all available access options and choose for themselves which one they want to use.25

Displaying all available services instead of using direct linking gives patrons the ability to access articles on the platforms they prefer, but it can become a problem when there are too many services to choose from. When the Alma link resolver identifies bibliographic records and electronic inventory that match the metadata elements in a Context Object, it returns and displays all relevant services in the Full Display and Services pages.26 This means that if there are portfolios for the same journal activated in multiple e-collections (as is often the case when there is overlap in aggregator subscription packages), a link resolver service for each of those portfolios will appear in the Full Display or Services pages. One article may have 10 or more services available, Which can be baffling for patrons who may not understand the differences between the services or Which one they should choose.

Bafflement can quickly turn into frustration if a patron chooses a service that results in a broken link, links to the journal instead of the article, or links to a completely different article. Because link resolver services are dynamic URLs, there is an increased risk of linking failure due to inaccurate or incomplete OpenURL metadata, incorrect knowledgebase holdings, or target parser issues.27

Enhancement Options

To address the aforementioned pain points and simplify the patron discovery -to-delivery workflow, the Systems and UX librarians investigated two direct linking methods during the initial phases of the project: Ex Libris's Quicklinks and Third Iron's LibKey Discovery.

Ex Libris's Quicklinks is an optional Primo feature that embeds direct links to the PDF and/or HTML full text in Brief Display records and Full Display pages for articles, books, and book chapters when a resource's CDI record contains sufficient metadata including a DOI or provider ID, the library has full text access rights to the resource, the activation source in Alma is an e- collection from a participating provider, and the e-collection has a CDI Collection ID.28 As of August 2024, approximately 60 providers support Quicklinks.29

LibKey Discovery is a discovery service integration in Third Iron's LibKey suite of products, which also includes LibKey Nomad, LibKey.io, LibKey Link, and BrowZine. LibKey products utilize an artificial intelligence-based linking mechanism that combines publisher metadata, information about the library's holdings, and the library's authentication method to create direct links to full text content at the best available source for the patron.30 The LibKey Discovery integration enables one-click access to the PDF and/or HTML full text for subscribed and Open Access journal articles from Brief Display records in search results and Full Display pages, as well as links to journal issues at BrowZine.31 The link labels and circumstances in which each link is displayed can be customized to suit the library's needs, and the technical documentation includes examples and code snippets for a variety of configurations.32 Like Quicklinks, LibKey Discovery links are only available for journal articles from publishers and platforms who supply metadata to Third Iron and meet certain technical requirements 33 As of January 2024, Third Iron supports access to journal content from over 5,000 scholarly publishers, and supported publisher titles can be linked to content hosted on 12 major platforms (such as EBSCO, ProQuest, and JSTOR) if the library subscribes to a supported title through the platform.34

Both direct linking options provide one-click, reliable access to full-text articles from Brief Display records in search results; however, when the TTU Libraries' Systems librarian conducted informal testing of Quicklinks' functionality and link availability, results found that Quicklinks did not display in Brief Display records very often. The infrequent availability of the links was most likely due to the fact that Quicklinks was a relatively new feature when the testing took place, and not many providers supported Quicklinks linking at that time. Third Iron's list of supported publishers and platforms was more extensive, therefore the TTU Libraries decided to evaluate direct linking with LibKey Discovery during a trial of Third Iron's LibKey suite of products.

IMPLEMENTING LIBKEY DISCOVERY

Integration Setup

The initial integration setup was completed by TTU Libraries' Systems librarian Barbara McArthur in September 2022. To establish the TTU Libraries' Third Iron account, Barbara completed a form with all the information Third Iron needed to set up the integration in their system, such as the TTU Libraries' IP ranges, EZproxy prefix, link resolver base URL, and URLs for the library website and ejournal search pages. Barbara then created a publishing profile in Alma which automatically sends the TTU Libraries' ejournal holdings to Third Iron on a weekly basis.

For the purposes of testing during the trial period, Barbara implemented LibKey Discovery in the TTU Libraries' Primo sandbox environment using instructions provided in Third Iron's technical documentation for LibKey Discovery. The technical documentation includes JavaScript code snippets for different configuration options so that libraries can choose which links to display in Primo, specify when the links display, and further customize the integration if needed; for example, one configuration option displays the links for all available full text formats plus the link resolver, and another will only display a link to the best full text format available and hide the link resolver when LibKey Discovery links are available.3> Barbara decided to use the configuration option that displays all available LibKey links in Primo Brief Display records for the initial implementation of LibKey Discovery, so she copied the corresponding code snippet from the technical documentation, applied some CSS changes to match the TTU Libraries' local preferences, and then pasted the customized version of the code snippet into the Primo Customization Package. Figure 2 depicts an example of an article Brief Display record and Full Display page with LibKey Discovery links as originally configured by the TTU Libraries.

Figure 2. The TTU Libraries initially configured LibKey Discovery to display article links for both the PDF and HTML full text, the link to the article's journal issue at BrowZine, and the "Online access" link resolver link.

Third Iron provides clear, easy to follow documentation for each step in the implementation process, and Barbara successfully completed the initial setup and all subsequent changes to the configuration without issue.

Initial Testing

After completing the initial setup and configuration of the integration in the Primo sandbox, Barbara and the UX unit completed an internal evaluation of both the functionality and usability of LibKey Discovery as well as other LibKey products included in the TTU Libraries' trial. Barbara recruited 12 internal testers from various departments of the TTU Libraries to participate in the LibKey Discovery functionality testing. Using a test script Barbara developed, the internal testers performed searches for a combined total of 10 articles and 12 journals in the Primo sandbox using keywords of their choice, then tested the LibKey links to full-text articles ("Download PDF" and "Read Article") as well as the links to view the journal or journal issue at BrowZine. The internal testers were asked to indicate if the LibKey Discovery links directed them to the appropriate resource and make note of how often the LibKey Discovery links displayed in the first 20 search results. None of the internal testers reported problems with linking accuracy. The LibKey Discovery links displayed in the first 20 search results for journals 25% of the time (without filtering for journals only) and displayed in the first 20 search results for articles 75% of the time.

After the functionality testing concluded, Barbara and the UX unit investigated the usability of the LibKey Discovery "Download PDF" and "Read Article" links and how they compared to Primo's linking methods in terms of ease of use. The UX unit conducted user tests with 12 library patrons in November 2022. Participants included students and faculty, and only half of the participants said they had used Primo before. During the sessions, participants were asked to perform a search in the Primo sandbox using a keyword related to their research interests, then use the LibKey Discovery links to either download or view two peer-reviewed articles. After the participants completed the tasks, the moderators asked them to share what was easy or difficult about using the LibKey Discovery links as well as any other feedback they'd like to give. Overall, the participants liked the integration and described it as "quick" and "simple." Participants who were familiar with Primo noticed that with the LibKey Discovery links, they were able to access articles more quickly and with fewer clicks. Two of the participants who had not used Primo before used the link resolver links to navigate to the first article they selected and were then surprised and pleased by how much more quickly they were able to access subsequent articles using LibKey Discovery links. While participant feedback was mostly positive, they disliked that notall of the articles in their search results had the LibKey Discovery links. They also expressed that there were now too many links on the Brief Display records, and they "wouldn't know what or which to click."

Following the positive feedback and insight gained, the TTU Libraries decided to move forward with LibKey Discovery after the trial ended in December 2022. The subscription began in January 2023, and Barbara prepared to launch LibKey Discovery in the TTU Libraries' Primo production environment by arranging a live demonstration for librarians and staff. Barbara repeated the configuration process from the trial in the Primo production environment, and LibKey Discovery officially launched in May 2023 after the end of the spring semester.

ASSESSING LIBKEY DISCOVERY

Following the successful implementation of LibKey Discovery in the Primo production environment, Systems and UX librarians began evaluating the integration to gauge its impact on the discovery-to-delivery workflow and identify potential areas of improvement. The assessments included an additional round of usability testing as well as an analysis of LibKey Discovery and link resolver usage statistics.

Usability

In October 2023, TTU Libraries' UX librarian Kimberly Vardeman and the UX unit conducted user feedback sessions focused on whether the link labels in the Brief Display record were clear and understandable and if any of the links were redundant or unnecessary. Nine undergraduate students participated in the sessions, six of whom indicated that they had used Primo to find research articles before. The participants were asked to perform a search in Primo using a keyword relevant to one of their courses, then download or view three peer-reviewed articles from the results. Moderators made note of whether the participants clicked on a LibKey Discovery link in the Brief Display record. If a participant did not choose any of the LibKey Discovery links, the moderators noted the steps they took to get to the article instead. Once the participants completed the process, the moderators asked participants if they felt the way the links were labeled was clear, how they might change the wording if they felt it was not clear, and how they would rate the usefulness of the links they used to download or view the articles.

Participants selected a total of 25 articles to view or download. Non-article selections were excluded from analysis. Participants used the "Download PDF" link to access articles a total of 15 times, regardless of the record display type, either Brief Display or Full Display. The "Read Article" link was used to access articles a total of four times; however, the "Download PDF" link was not available for one of those articles, and the participant who selected it had used the "Download PDF" link to access the other two articles they selected. None of the participants clicked on the "View Issue Contents" link, as it did not directly relate to this task, and participants used a link resolver service instead of a LibKey Discovery link to access six articles.

Interestingly, participants chose to click on the article title and open the article's Full Display page rather than go directly to the article using the LibKey Discovery links in the Brief Display record for 14 of the 25 articles. The UX librarian hypothesized that users took this less-direct path because they either did not see the download links or were more accustomed to Google and other websites where clicking the title pointed them directly to the resource. In addition, instruction librarians had taught students to view additional information about the resource, e.g. abstract, summary, database platform, by clicking the title link, which might have explained the behavior of some participants. Notably, the three participants who had never used Primo did not click the article title-they all used the Download PDF option to access their initial article selection. That observation could indicate that users who were already familiar with Primo followed their existing navigation patterns instead of opting to use the new features. The true explanation is not evident from this study. Among participants who clicked on the title initially, the participants selected the "Download PDF" link in the Full Display record six times, and the "Read Article" link two times. Participants used the "Download PDF" link in the Brief Display record to access nine articles, and the "Read Article" link in the Brief Display record was only used to access two articles.

When asked about the clarity of the wording used for the links' labels, participants generally thought that the links were "clear" and "pretty straightforward"; however, there was some confusion over the "Online Access" link, and two of the participants asked what "View Issue Contents" meant. Two participants stated that the links blended in with the rest of the text on the records and did not stand out to them and suggested that changes to the font size and color would make the links more noticeable. In response to questions about the usefulness of the LibKey Discovery links, the participants stated that the "Download PDF" and "Read Article" links reduced the number of clicks required to navigate to full-text articles, and that the links would make accessing articles easier.

After the user feedback sessions concluded, the UX unit reviewed their findings and recommended that specific aspects of the integration's configuration be adjusted in response to the participants' feedback. Because the feedback regarding the "Download PDF" and "Read Article" links was very positive overall, the UX unit recommended that both links should be retained, but that the font color, size, and style be simplified for the LibKey Discovery links as well as the "Online Access" link. The UX unit also recommended that the "View Issue Contents" link be removed from the integration's configuration per feedback received during both rounds of user feedback sessions. Participants from the initial user feedback sessions felt that there were too many links at the bottom of the Brief Display records, and the visual clutter potentially distracted users from their primary goal of obtaining the article full text.

Per the UX unit's recommendations, Barbara modified the LibKey Discovery configuration so that the "View Issue Contents" link no longer displays on article Brief Display records or Full Display pages and adjusted the TTU Libraries' local CSS customizations for the link options' font styles.

Usage

To supplement the insight gained from the user feedback sessions, TTU Libraries' Electronic Resources Librarian Jessie London conducted usage analyses which were intended to (1) verify that patrons were in fact using the LibKey Discovery integration and establish a baseline to which future usage could be compared, (2) assess LibKey Discovery's impact on how patrons used Primo to access full-text articles, and (3) compare patron feedback to patron behavior (i.e., what patrons say versus what patrons actually do when they are not being observed or asked to share their opinions).

Data sources for the analysis included Third Iron usage reports for May 2023 through April 2024 and reports created in the Link Resolver Usage subject area of Alma Analytics. The Third Iron usage reports are delivered to Systems librarians at the TTU Libraries via email on a monthly basis, and the usage data is provided in an Excel file with separate sheets for each section of the report. The Summary section contains usage data for full text download requests recorded by all Third Iron services combined as well as each individual feature, plus metrics specific to LibKey Nomad and BrowzZine.36 Figure 6 depicts an example of the Summary section. The My Bookshelf and TOC (Table of Contents) Views sections report the My Bookshelf and journal table of contents views recorded in BrowZine.37 The Full Text Download Requests section provides the number of full text download requests initiated via all Third Iron services, broken out by journal.38 Reports delivered prior to April 2025 included an Overview section which contained a link to the usage statistics documentation available in the Third Iron Support Knowledge Base; however as of March 27, 2025, the link is now provided in a Help section.39 Most relevant to this analysis is the Summary section's "LibKey Discovery and API Services" metric, which indicates the total number of full-text downloads requested from the LibKey Discovery integration in discovery services or other third-party services which use the LibKey API such as Leganto, InfoDesk, PICO Portal, and others.40 The TTU Libraries are only using LibKey Discovery in Primo, so it was not necessary to take API services usage into account when analyzing this data.

In total, 117,407 full-text downloads were initiated using the LibKey Discovery integration with Primo during the first year following its implementation in production. Usage was predictably low during the first month due to the timing of the implementation, but it did not stay low for long. October 2023 had the highest usage with 15,186 full-text downloads. While there was a sharp decrease from November 2023 to January 2024, usage picked up again once the Spring 2024 semester started.

The data included in Third Iron's usage reports is very useful, but it's difficult to determine the integration's impact on access to full-text articles from Primo with this data alone. To contextualize the LibKey Discovery usage data and assess link resolver usage both pre- and post-implementation of LibKey Discovery, custom Alma Analytics reports were created in the Link Resolver Usage subject area for May 2020-April 2021, May 2021-April 2022, May 2022-April 2023, and May 2023-April 2024.

The Link Resolver Usage subject area includes various metrics for assessing how the Alma link resolver is used to access e-resources, such as the number of OpenURL requests sent to the link resolver, the number of times users clicked on at least one of the link resolver services returned by arequest, and the number of requests that did not return any services. The metrics used in the reports included the "Number of Requests," "Number of Clicked Requests," and "Number of Requests Without Services" from the OpenURL Context Measures fact table. OpenURL requests are sent to the Alma link resolver when a user opens a Full Display page, whether it be by clicking on the title or "Online Access" link in Brief Display records; or when a user lands on a Services page after clicking on an Alma link resolver link at an external source such as Web of Science, EBSCO, or Google Scholar.41 A link resolver "request" is recorded when Alma receives an incoming OpenURL request. A "clicked request" (i.e., clickthrough) is recorded when a user clicks on at least one of the services available in a Full Display or Services page in order to access the full text of a resource; conversely, when an incoming OpenURL request does not return any services, a "request without services" is recorded.·" The "Number of Requests Without Services" metric was used in conjunction with the other measures to calculate the number of link resolver requests received that returned services, but no clickthroughs were recorded.

Multiple filters were applied to the Analytics reports in order to return data relevant to the analysis. Since the LibKey Discovery integration only provides links to journal articles, the reports were limited to requests with an OpenURL Context Object material type of "JOURNAL" that also had article-level metadata present in the OpenURL. To exclude Course Reserves usage as well as usage attributed to staff testing from Alma, all requests with OpenURL Context Object sources indicating that the service available was for an Alma record accessed from either Primo or Alma were filtered out of the reports. No other sources were excluded. An example of the SQL code issued for the Alma Analytics link resolver usage reports is included in appendix A.

As shown in figure 8, the number of link resolver clickthroughs for journal articles recorded per month between May 1, 2023, and April 30, 2024, was significantly lower than the number of clickthroughs recorded during the same period in previous years.

Ex Libris does not track usage of third-party integrations, so link resolver usage alone is no longer an accurate representation of patrons' use of Primo for access to full-text articles. Access to journal articles was primarily facilitated by the Alma link resolver prior to the implementation of LibKey Discovery, so a comparison of link resolver and LibKey Discovery usage was necessary to verify that the drastic decrease in link resolver usage was mainly due to LibKey Discovery and not other, more alarming possibilities (e.g., patrons simply stopped using Primo). Figure 9 shows that the number of link resolver and LibKey Discovery clickthroughs recorded per month from May 1, 2023, to April 30, 2024, remained relatively close, but link resolver usage exceeded LibKey Discovery usage for most of the year. In January 2024, LibKey Discovery usage began to exceed link resolver usage. This shift could potentially be attributed to an increase in the number of search results with LibKey Discovery links available; but it's also possible that more patrons began to notice the LibKey Discovery links after the UX unit's recommended changes to the integration were completed.

When combined, the total number of link resolver and LibKey Discovery clickthroughs recorded from May 1, 2023, to April 30, 2024, was similar to the number of link resolver clickthroughs recorded in previous years. Since patrons' overall use of Primo for access to full-text articles only slightly decreased during the first year following the implementation of LibKey Discovery, the decrease in link resolver usage depicted in figure 8 was not considered cause for alarm. Rather, the decrease in link resolver usage suggests that the LibKey Discovery integration has improved user experience by decreasing the number of clicks required to access full-text articles and increasing patrons' efficiency.

A significant decrease in the total number of link resolver requests recorded between May 1, 2023, and April 30, 2024, supports this finding and illustrates how LibKey Discovery has impacted the ways patrons interact with the Primo interface. Clicking on a LibKey Discovery link completely bypasses the link resolver, so the overall decrease in link resolver requests received is most likely due to patrons opting to use one of the LibKey Discovery links to access articles directly from Brief Display records instead of opening Full Display pages.

There was also an increase in the number of link resolver requests that had services available, but no clickthroughs were recorded. The increase in the number of requests with no clickthroughs may be explained by users clicking on an article title in the search results and selecting a LibKey Discovery link from the Full Display page instead of the Brief Display record. Clicking on an article title in a Brief Display record sends an OpenURL request to the Alma link resolver, but a clickthrough is only recorded if a link resolver service is selected from the View It menu in the Full Display page. The UX unit noted during the October 2023 user feedback sessions that participants clicked on a LibKey Discovery link in Brief Display records for 44% of articles selected and on the article title for 56% of the articles. A little over half of the participants who clicked on an article title ultimately used a LibKey Discovery link to navigate to the article instead of a link resolver service. Similarly, the number of requests with no clickthroughs represents roughly half of all requests with services available recorded after implementing LibKey Discovery. In previous years, the number of clickthroughs greatly exceeded the number of requests with no clickthroughs. Limitations

The findings from the usability testing and the analysis of LibKey Discovery and link resolver usage data provided valuable information regarding LibKey Discovery's impact on delivery of fulltext articles; however, there are still some unanswered questions due to the assessments' limitations.

Only usage data recorded during the first year of LibKey Discovery's integration with Primo was utilized in this analysis. To identify distinct trends and use patterns, LibKey Discovery usage statistics will need to be continually evaluated.

Because Ex Libris does not track usage of third-party integrations, an exact one-to-one comparison of Primo's linking methods (i.e., the Alma link resolver and CDI Link in Record links) and LibKey Discovery usage is not possible. The Alma Analytics reports utilized in the analysis were limited to requests with an OpenURL Context Object material type of "JOURNAL" and article-level metadata present in the OpenURL, but incongruities in bibliographic metadata in Alma, resource types in Primo, and CDI resource types may have resulted in OpenURL requests for magazine, newsletter, newspaper, and trade journal articles being included in the reports. Even if non-journal content is published by a provider or platform supported by Third Iron, LibKey Discovery links are not available for these resources due to a lack of metadata availability.·3 CDI Link in Record links do not invoke the link resolver, so any usage attributed to this linking method was notincluded in the analysis. While the odds of a scholarly journal article record providing a CDI Link in Record link instead of link resolver services are low due to local configurations, it would not be accurate to say that it never happens.

It is also worth noting that clickthroughs (regardless of linking method) do not necessarily mean that patrons were able to successfully access full-text articles. Functionality testing completed internally by TTU Libraries' personnel during the LibKey trial indicated that LibKey Discovery provides excellent rates of linking accuracy, but more extensive testing of link resolver and Link in Record linking functionality would result in a more robust assessment of linking options available in Primo.

Finally, the link resolver usage data alone cannot be used to determine why patrons choose one link over another, and the user feedback sessions did not specifically address participants' reasoning behind the links they chose. Systems and UX librarians expected LibKey Discovery usage to overtake link resolver usage soon after the integration launched, but link resolver usage remained higher throughout most of the year. It's possible that link resolver usage remained higher than LibKey Discovery usage because LibKey Discovery links were not available for the articles patrons wanted to access. It could also be that some patrons just prefer the link resolver or used link resolver services out of habit, or they simply did not notice the LibKey Discovery links in article Full Display pages if they clicked on the article title instead of a LibKey Discovery link in Brief Display records. Another possibility is that patrons expect to be taken to the full text when they click on an article title, just as they would if they clicked on a Google search result. Additional user feedback sessions with a larger group of participants would provide a better understanding of user behaviors, link preferences, and expectations.

DISCUSSION

Integrating LibKey Discovery with the TTU Libraries' Primo environment has resulted in significant changes in how patrons use Primo to access full-text articles. While Ex Libris offers one-click access options that are built into Primo, concerns regarding OpenURL direct linking reliability and insufficient Quicklinks coverage were major factors in the TTU Libraries' decision to implement LibKey Discovery. LibKey Discovery offers patrons accurate, one-click access to articles while also still giving them the option to access articles on other platforms using link resolver services if that is their preference. Based on usage statistics, patrons quickly incorporated LibKey Discovery links into their workflows, indicating that LibKey Discovery links offered a better, more intuitive user experience. The implementation process was straightforward, and the integration does not require any more maintenance than Primo's built-in one-click access methods. Because the TTU Libraries' ejournal holdings are automatically sent to Third Iron on a weekly basis via a publishing profile, updates to the TTU Libraries' holdings data at Third Iron do not require staff intervention. As long as the ejournal holdings are up to date and accurate, the LibKey Discovery links will be accurate. There have been no reports of LibKey Discovery links not working or taking patrons to incorrect articles. Any instances in which a patron encountered a paywall after using a LibKey Discovery link were due to holdings inaccuracies in Alma or platform issues and not issues with LibKey's linking mechanism. LibKey Discovery has also improved access to Open Access (OA) articles in both fully OA and hybrid journals, regardless of the journals' activation status in Alma.·· Links to OA articles are sourced from a combination of Third Iron-curated knowledgebases, Unpaywall data, and other sources, which allows LibKey Discovery to provide the best link available for access to an OA version of an article.43

LibKey Discovery certainly adds value to the TTU Libraries' Primo discovery environment, but it does have some limitations. The integration is currently only capable of providing direct links to scholarly journal articles available from content providers that supply Third Iron with article-level metadata (including DOIs and PMIDs) and provide articles in PDF and/or HTML format.46 LibKey Discovery links are not available for non-scholarly journal articles or other resource types, and the links will not display in records for scholarly journal articles if they are not available from a supported publisher or platform, a DOI or PMID is not present in the article-level metadata, or the article itself has not been assigned a DOI or PMID. In addition, the "Download PDF" links are only available for articles from content providers that support LibKey's Direct to PDF linking, and LibKey Discovery will only display the "Read Article" link for any articles from content providers that do not support Direct to PDF linking.47 The lack of consistency in link availability and whether one or both LibKey Discovery links displayed resulted in confusion for some participants in the user feedback sessions.

Assessments of LibKey Discovery's usability and usage data recorded during the first year following its implementation provided key insights regarding the integration's value. Participants in the UX unit's user feedback sessions overall felt that LibKey Discovery would make it faster and easier for them to access full-text articles, and some patrons have even expressed their appreciation for the LibKey Discovery links directly to Systems librarians. However, some participants felt that the LibKey Discovery links weren't noticeable, and others didn't understand why the links were only available for some of the articles in their search results. The usage data supports what the Systems and UX librarians have heard from patrons, and while usage was not quite as high as expected, findings from the user feedback sessions and analysis of link resolver usage indicate that LibKey Discovery usage may increase if the links are addressed during instruction sessions and reference consultations. The TTU Libraries launched LibKey Discovery with very little fanfare, and as the UX unit observed during the user feedback sessions, patrons may notimmediately notice the LibKey Discovery links. Due to Google's ever-present influence, most patrons are accustomed to clicking on search results in order to access the information they're seeking. Patrons new to Primo may expect to be taken to the full text after clicking on search results, and those who are familiar with Primo may still be clicking on an article title in order to view its available linking options located in the middle of Full Display pages. The LibKey Discovery links display at the top of Full Display pages, so they are easy to miss if patrons don't know to look for them.

Introducing LibKey Discovery to patrons as part of instruction sessions would give the TTU Libraries' subject librarians an opportunity to promote LibKey Discovery and mitigate patron confusion over why some articles have the links and others don't. Going forward, systems librarians will continue to monitor LibKey Discovery and link resolver usage for emerging trends and the UX unit will conduct additional usability studies in order to identify potential improvements.

CONCLUSION

The LibKey Discovery integration with Primo has enhanced linking to full-text articles and simplified the discovery-to-delivery workflow for TTU Libraries patrons. Patrons had previously expressed dissatisfaction with Primo's linking to full-text articles, specifically that getting from search results to the full text of an article required too many clicks, there were often too many links to choose from, and sometimes the links didn't work or directed patrons to the wrong resource. LibKey Discovery saves patrons a minimum of two clicks by taking them directly to the article PDF or HTML full text from Primo search results using LibKey's linking technology, thus improving both usability and linking accuracy. Patron feedback and usage data collected during the TTU Libraries' first year with LibKey Discovery indicated an overall positive response to the integration, and usage is expected to increase as more subject librarians incorporate LibKey Discovery in their instruction sessions. LibKey Discovery has undoubtedly added value to Primo, and the TTU Libraries expect that the integration will continue to have a positive impact on user experience.

Sidebar

About the Authors

Submitted: 9 October 2024. Accepted for Publication: 18 April 2025. Published: 16 June 2025.

Footnote

ENDNOTES

1 David Wells, "Online Public Access Catalogues and Library Discovery Systems," Knowledge Organization 48, no. 6 (2021): 457-66, https: //doi.org/10.5771/0943-7444-2021-6-457.

2 Marshall Breeding, "Index-Based Discovery Services: Current Market Positions and Trends," Library Technology Reports 54, по. 8 (2018), https: //doi.org/10.5860/1tr.54n8.

3 Marshall Breeding, "Next-Generation Library Catalogs," Library Technology Reports 43, no. 4 (2007): 38-41, https: //doi.org/10.5860/1tr.43n4.

4 Jason Vaughan, "Web Scale Discovery Services," Library Technology Reports 47, no. 1 (2011), https://doi.org/10.5860/1tr.47n1.

5 Breeding, "Index-Based Discovery Services."

6 Aaron Tay, "Improving Access to and Delivery of Academic Content from Libraries," Library Technology Reports 58, no. 6 (2022): 1-47, https://doi.org/10.5860/ltr.58n6.

7 Herbert Van de Sompel and Oren Beit-Arie, "Open Linking in the Scholarly Information Environment Using the OpenURL Framework," D-Lib magazine 7, no. 3 (2001), https://www.dlib.org/dlib/march01/vandesompel/03vandesompel.html.

8 Kenyon Stuart, Ken Varnum, and Judith Ahronheim, "Measuring Journal Linking Success from a Discovery Service," Information Technology and Libraries ЗА, no. 1 (2015): 52-76, https://doi.org/10.6017 /ital.v34i1.5607.

9 Ashley Zmau and Holly Talbott, "The Current Landscape of Electronic Resources Access Issues," Library Technology Reports 58, no. 7 (2022), https: //doi.org/10.5860/1tr.58n7.

10 Scott Hanrath and Miloche Kottman, "Use and Usability of a Discovery Tool in an Academic Library," Journal of Web Librarianship 9, no. 1 (2015): 1-21, https://doi.org/10.1080/19322909.2014.983259.

11 Hanrath and Kottman, "Use and Usability of a Discovery Tool," 1-21.

12 Alexandra Hamlett and Helen Georgas, "In the Wake of Discovery: Student Perceptions, Integration, and Instructional Design," Journal of Web Librarianship 13, no. 3 (2019): 230-45, https: //doi.org/10.1080/19322909.2019.1598919.

13 Kerry Walton, Gary M. Childs, and Laurie Palumbo, "Testing Two Discovery Systems: A Usability Study Comparing Student Perceptions of EDS and Primo," Journal of Web Librarianship 16, no. 4 (2022): 200-221, https: //doi.org/10.1080/19322909.2022.2125478.

14 Walton, Childs, and Palumbo, "Testing Two Discovery Systems," 200-221.

15 Breeding, "Index-Based Discovery Services."

16 Andrew R. Clark et al., "Taking Action on Usability Testing Findings: Simmons College Library Case Study," The Serials Librarian 71, no. 3-4 (2016): 186-96, https://doi.org/10.1080/0361526X.2016.1245170.

17 Stuart, Varnum, and Ahronheim, "Measuring Journal Linking Success," 52-76.

18 Stuart, Varnum, and Ahronheim, "Measuring Journal Linking Success," 52-76.

19 Kate Silton, "Assessment of Full-Text Linking in Summon: One Institution's Approach," Journal of Electronic Resources Librarianship 26, no. 3 (2014): 163-69, https://doi.org/10.1080/1941126X.2014.936767.

20 Jill K. Locascio and Dejah Rubel, "A Tale of Two Tools: Comparing LibKey Discovery to Quicklinks in Primo VE," Information Technology and Libraries 42, no. 2 (2023): 1-6, https://doi.org/10.6017 /ital.v42i2.16253.

21 "Introduction to Alma-Primo Integration," Ex Libris Knowledge Center, Ex Libris, 2025, https: //knowledge.exlibrisgroup.com/Alma/Product Documentation/010Alma Online Hel English)/060Alma-Primo Integration/010Introduction.

22 "CDI and Linking to Electronic Full Text," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Primo/Content Corner/Central Discovery Index/Docu mentation and Training/Documentation and Training (English)/CDI - The Central Discovery Index/050CDI and Linking to Electronic Full Text#.

23 "Alma Link Resolver Workflow," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Alma/Product Documentation/010Alma Online Hel English) /Electronic Resource Management/051Link Resolver/010Alma Link Resolver Work flow.

24 "CDI and Linking to Electronic Full Text."

25 "View It: Configuring Direct Linking," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Alma/Product Documentation/010Alma Online Hel English)/030Fulfillment/080Configuring Fulfillment/090Discovery Interface Display Logic 035Viewlt#Configuring Direct Linking.

26 "Alma Link Resolver Workflow."

27 Tay, "Improving Access to and Delivery of Academic Content from Libraries," 1-47.

28 "Quicklinks in Primo and Primo VE," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Primo/Product Documentation/020Primo VE/Primo V E (English) /030Primo VE User Interface/Quicklinks in Primo and Primo VE.

29 "Quicklinks Provider List," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Primo/Product Documentation/020Primo VE/Primo V E (English) /030Primo VE User Interface/Quicklinks Provider List.

30 "LibKey FAQ for Publishers," Third Iron Support, Third Iron, 2025, https: //support.thirdiron.com/support/solutions/articles /72000575804-libkey-fag-for-publishers.

31 "Ex Libris Primo Integration," Third Iron Technical Documentation, Third Iron Support, 2024, https:/ /thirdiron.atlassian.net/wiki/spaces /BrowZineAPIDocs /pages/79200260/Ex+Libris+P rimo+Integration.

32 "Ex Libris Primo Integration."

33 "What Are the Requirements to Include a Journal in LibKey and BrowZine?", Third Iron Support, Third Iron, 2024, https: //support.thirdiron.com/support/solutions/articles/72000575802-what-are-the-requirements-to-include-a-journal-in-libkey-and-browzine-.

34 "What Publishers Do You Support?", Third Iron Support, Third Iron, 2024, https: //support.thirdiron.com/support/solutions/articles /72000575799-what-publishers-do-you-support.

35 "Ex Libris Primo Integration."

36 "Third Iron Usage Statistics," Third Iron Support, Third Iron, 2025, https:/ /support.thirdiron.com /support/solutions/articles/72000570016-third-iron-usage-statistics.

37 "Third Iron Usage Statistics."

38 "Third Iron Usage Statistics."

39 "Third Iron Usage Statistics."

40 "Third Iron Usage Statistics."

41 "Alma Link Resolver Workflow."

42 "Link Resolver Usage," Ex Libris Knowledge Center, Ex Libris, 2024, https: //knowledge.exlibrisgroup.com/Alma/Product Documentation/010Alma Online Hel English) /080Analytics/Alma Analytics Subject Areas/Link Resolver Usage#.

43 "What Publishers Do You Support?"

44 "Open Access Linking and Configuration Options," Third Iron Support, Third Iron, 2022, https:/ /thirdiron.freshdesk.com/support/solutions/articles /72000569332 -open-access-linking-and-configuration-options.

45 "Open Access Linking and Configuration Options."

46 "What Are the Requirements to Include a Journal in LibKey and BrowZine?"

47 "Publishers Supported by LibKey Direct to PDF Linking," Third Iron Support, Third Iron, 2023, https:/ /support.thirdiron.com/support/solutions/articles/72000569839-publishers-supported-by-libkey-direct-to-pdf-linking.