Use and users of the Minrva mobile app

Abstract

Purpose

The purpose of this paper is to investigate new undergraduate student library engagement in the Minrva mobile app during the months of May 2015 through December 2015.

Design/methodology/approach

This research investigated what parts of a mobile app new students were using in their first semester after downloading the app. The quantitative study used application programming interface log analysis to better understand what parts of the app new students use in the mobile app.

Findings

By undertaking this study, the author has a better understanding about what students are finding useful within the app and what tools are not being used by this cohort in their first semester.

Originality/value

The value of this research is in helping system designers and first-year experience planners know what mobile support tools students are finding useful in their first semester. Implication for mobile interface design based on module popularity are discussed.

Introduction

Outreach to new undergraduate students with promotion of help guides, resources and library expertise focused on supporting first-year student transitions is an important and ongoing concern for undergraduate focused academic libraries. Analysis into how new students interact with the library and the effects of this interaction on academic performance and degree persistence with multi-dimensional data (e.g. learning management systems, computer lab use and library uses) analysis has become popular in the literature of student learning (Soria et al. , 2014, 2017). Learning analytics was defined in the Horizon Report's 2016 Higher Education Edition, as "an educational application of web analytics aimed at learner profiling, a process of gathering and analyzing details of individual student interactions in online learning activities" (Johnson et al. , 2016). Learning analytics represents recent interest by librarians and higher education administrators to explore student learning data generated online (Long and Siemen, 2011). Learning analytic studies are now possible in large part due to the fact that much of student learning activity happens online. Within the broader context of higher education, by strategizing for learning analytic technologies and assessments, preemptive interventions could be implemented to support degree persistence. Other perceived benefits to undertaking learning analytic studies include providing individual students a personal learning experience by tailoring pedagogies to individuals within education technology platforms (Johnson et al. , 2016). These efforts underscore a broad concern for student success by analyzing in detail the digital touch points of students through personal data that are generated in the online learning ecosystem. This study is a related quantitative analysis into understanding what resources students are making use of within a focused platform: the library mobile application Minrva. Researchers sought specifically to investigate usage of the app during the first semester of study.

The Minrva mobile app within new student orientation

The University of Illinois Undergraduate Library heavily promotes the Minrva app at orientation events during the summer months and during the first weeks of the fall semester[1]. The promotions specifically target new undergraduate students, which includes transfer students as well as new first-year students. In the months leading up to the Fall 2015 semester, an unusually high number of downloads were recorded for the app. The details of total downloads are included in the results section of this paper. The motivations to look at server logs of the application was driven by the awareness that many more students were downloading the app than usual, and librarians wanted to understand what parts of the app were the most popular after observing many downloads. It was hypothesized that by looking at the server logs themes of popularity might emerge and inform librarians what students actually find interesting and useful from a library mobile application. By looking to understand popular parts of the app, librarians can better understand what students need and what they actually use in their first semester of study. The Minrva mobile app is a way that the library can provide ongoing orientation to services, collections and expertise. The Minrva mobile app was developed modularly so that app functions are separated into different parts of the app (Figure 1).

App modularity helps to support overall app stability and makes it easy to add or remove functionality within the app over time. The 1.0 version of the Minrva mobile app was first released in 2012 (Hahn and Ryckman, 2012). With a series of focused research grants the Minrva app functionality was increased to encompass a set of core modules and newly developed custom modules leveraging emerging technology which the user can select based on their library location (Hahn, 2015; Hahn et al. , 2015). Minrva has had three major releases from 2012-2016. As of October 2016, the Minrva app reached version 3.0. At the time of this study, the University of Illinois had developed and made available within the app the following core modules (Ryckman and Ryckman, 2016):

* Blocks : The blocks module displays a list of blocks against this account.

* Catalog search : The catalog module can search a single I-Share library location or all I-Share libraries at once. The location can be set on the home screen.

* Checked out : Displays a list of checked-out items. Users can renew any of their currently checked-out items.

* Citation : Generates a citation for a library item. Generated citations can be copied or sent through an e-mail using an e-mail client on the phone.

* Course reserves : The course reserves module searches for all course reserve materials by instructor, course or department.

* Favorites : The favorites module displays a list of items that a user has marked as a favorite.

* Fines : Displays a list of all fines associated with this account.

* Requested items : Displays a list of all of the items that a user has requested.

* Updates : The Minrva updates module displays updates related to the Minrva app.

The app also includes several custom modules that are provided to the student based on their library location. The custom modules available during the Fall 2015 semester include (Ryckman and Ryckman, 2016):

* Best sellers : The best sellers module displays bestselling books, available for checkout through the University of Illinois' libraries.

* Events : The events module displays a complete list of current news about upcoming library events, exhibits and activities.

* Hours : This displays which of the University of Illinois Libraries are currently open or closed.

* Journal search : The journal search module searches a wide range of databases for online articles.

* Question board : Question board is an anonymous question and answer service provided by the Undergraduate Library at the University of Illinois.

* Scanner : The scanner module scans the library barcode and is able to call information about the scanned item.

* Technology : The technology module displays a list of rentable technology and the number of each item that is still available.

* Topic space : Topic space allows a user scan an item's barcode in the book stacks. The module will show the user other books that are relevant but that are not shelved nearby.

* Wayfinder : The wayfinder module displays the location of the most recently searched and selected item.

Detailed screenshots and functionality of these modular components can be found on the Minrva Project website (). Each module has a corresponding web service or application programming interface (API)[2]. Because each module depends on one API to function, web server analytics can be used to analyze the module uses after download.

Research questions

The goal of this research is to come to a better understanding of specifically what happens within the Minrva mobile app after a high number of new students download the app. To understand this use server logs of the app were parsed and analyzed for module-specific API hits. The time period under study included the first day of transfer student orientation, May 1, 2015 until the end of the calendar year, December 31, 2015. Specifically, this research is focused on identifying most popularly used modules within the Minrva application and also understand if there are any modules which students are not accessing.

This paper progresses next with a review of relevant literature on first year-student support, APIs and transaction log studies. Following this a methodology section details the study mechanics. In the results section, data analysis is presented detailing monthly hits indicating popular and less popular Minrva modules. Finally, in the paper's discussion and conclusion section findings are interpreted through the lens of addressing the first-year student uses of the Minrva mobile app and implications for future studies.

Literature review

New students in their first semester at university require targeted academic and social support. This new student support has manifested on campuses throughout higher education in several forms including learning communities, residential communities and through targeted campus orientation programs. Online orientation is an important way to reach new students, with higher education researchers noting that this model has proven to be cost effective (Brown and Hernandez, 2010). Indeed, the online mode of orientation has seen uptake specifically for transfer students because practitioners observed less attendance by transfer students for in-person orientation activities (Brown and Hernandez, 2010). Previous library-related studies new student experience programming has advocated for the development of new technology tools to rethink or redesign outreach activities. Development projects within higher education included iPod tours of the library (Mikkelsen and Davidson, 2011) as well as a variety of digital gaming initiatives (Fitz-Walter et al, 2011). But while the virtual experience of the student for online orientation is valuable for scaling resources and serving far greater numbers than in-person orientation could attend to, the on-site campus experience of students is important to consider once students begin courses. This is especially true considering that there is a range of preferences among the new student cohort; first-year students especially value in-person communication (Brown and Hernandez, 2010).

A mobile app design competition investigated how students would design apps to support their academic experience. Key findings of this case competition underscored space design implications when considering mobile apps and the student's academic life. For example, researchers found that:

"Location specific needs included the desire to identify key available resources in each facility, as well as characteristics of the spaces themselves", and that "discovery of campus learning spaces was another key theme for the groups. Specifically, one desired characteristic for some apps was the ability to recommend an alternate space if the space they were currently working in was not appropriate for them or their group. The app could also help with discovery of resources in a student's current location - providing information about features of their current space (such as reserveable study rooms) that they might not know about. It might then visually prompt them to take an action related to these features (such as booking a study room)" (Ward, Hahn, and Mestre, 2015, p. 35).

I also know through empirical research into the first-year experience of undergraduate students that course attendance by new students is an important indicator of success (Upcraft et al. , 2005, p. 62). Student success researchers have also advocated increased training of faculty to "incorporate activities to track attendance and identify students who could benefit from early interventions" (Greenfield et al. , 2013, p. 48).

Libraries interested in evaluating app usage have looked into comparing downloads of mobile apps as compared to a website usage (Wong, 2012). As it relates to mobile app analytics in library settings, it was previously reported that mobile app uses and website use was about the same for a video app in a University in Hong Kong (Wong, 2012). Findings from large scale studies of mobile application usage that the average use of an app is usually less than one minute - so the types of engagement that students will engage with an app is fairly limited in time (Böhmer et al. , 2011). The research group that studied mobile application usage also looked into the time of day and type of app used during the day. Their findings indicated that "news applications are most popular in the morning and games are at night, but communication applications dominate through most of the day". The authors go on to note that "We also find that despite the variety of apps available, communication applications are almost always the first used upon a device's waking from sleep" (Böhmer et al. , 2011, p. 47). This research was conducted by monitoring app activity within the mobile device. There are several approaches for doing this - with specific application monitoring tools across the entire device or targeting one specific application by looking at analytics incorporated as part of its operation, by way of data providers or within the app itself.

Application programming interfaces and transaction log analysis

The API is the foundational data source from which an app gets its data to display. At the time of this writing, several large internet companies are making platform services available to mobile apps (and other internet-based software) by way of web-based APIs. Take for example the Google Cloud Platform, which makes available several compelling APIs for valuable services that can be incorporated into mobile apps. These services include the speech API[3], which provides a highly accurate service that generates text from spoken word. Note also the availability of the Google Cloud Platform's Vision API, which provides a profoundly impactful image analysis service all by way of a RESTful web-based API[4]. With such a service images and text in images are analyzed and identified using state of the art machine learning technologies that form the back-end of the vision API.

Turning now to services for analyzing how library data are being used in the app, Firebase Analytics is a new software tool that helps mobile developers analyze (among other uses) how their APIs are being consumed[5]. The tool is a welcome addition to the mobile developer's toolkit for analyzing usage trends at a very detailed level. At the time of this study, Firebase Analytics had not yet been incorporated into the Minrva app, but it is planned to be implemented in the 3.1 release to provide more fine-grained analytics and interface assessment data.

According to a recent scholarly analysis of a popular developer discussion forum, researchers underscored the importance of APIs specific to mobile app development noting:

Developers and companies should focus more on its APIs design and thinking on how it will be consumed to help developers. Establishing and documenting the APIs services should be one of the first tasks in creating software today, establishing a contract so that developers can quickly learn about services offered and give feedback early (Rosen and Shihab, 2015, p. 1205).

A code4lib journal article on mobile development similarly indicated the importance of APIs (also referred to as web services) for mobile development (Galvin and Sun, 2011).

There is a rich research area on search log analysis of library discovery tools and web searches (Peters, 1993; Mischo et al. , 2012; Lown et al. , 2013). However, a targeted search in two major Library and Information Science databases (Library and Information Science Source, Library and Information Science Abstracts) for the terms "API log analysis" did not yield search results with any prior studies relevant to this research study. Therefore, the research methodology utilized to understand student use and users within academic libraries appears to be unique with regard to mobile use analysis by way of API logs though this study is certainly related to previous research studies in the Library and Information Science literature on log analysis (Peters, 1993; Jansen, 2006).

Jansen's paper in particular is instructive in carrying out this type of research, referred to in the literature as Transaction Log Analysis (2006). Jansen defines Transaction Log Analysis as:

[...] the use of data collected in a transaction log to investigate particular research questions concerning interaction among Web users, the Web search engine, or the Web content during searching episodes (Jansen, 2006, p. 409).

There are three key parts of log analysis: collection, preparation and analysis (Jansen, 2006). The APIs that send data into the Minrva app recorded standard webserver logs of user interactions with the app. Due to the standardized format, the preparation needed to undertake the analysis was simple and only required a trivial amount of cleanup processes that allowed for Minrva module analysis - detailed next in the methodology section.

Methodology

The time intervals that were the focus of inquiry for the log analysis was from May 1, 2015, the first day of transfer student orientation at the University of Illinois Campus, until December 31, 2015. The author gathered all the download data for Minrva by way of both Android and iOS app store administrative interfaces. Next, webserver logs from the Minrva server were analyzed using Splunk Enterprise analytics. Splunk Enterprise is a tool commonly used to analyze server events. In this research, it was used to filter server events so that an analysis of API events from the server to the user's mobile phone could be analyzed.

Preprocessing or preparation was the main focus of the Splunk software tool. The first steps to preparation included removing very large access hits from IP ranges that were considered to be from automated sources. These sources were most likely programs run for other purposes and would not represent actual student uses. After removing several highly accessed API outliers there were 45,975 API hits for the study. Removal of the outliers gives us a more accurate picture of app use because these outliers are likely from automated sources.

To undertake the analysis of prepared data, the project researcher looked at each month of API log hits and annotated the API hits to their corresponding Minrva module. All pre-processed and annotated data are available in the Illinois Data Bank, a public repository for publishing research data from the University of Illinois at Urbana-Champaign (Hahn, 2016). The personally identifiable data points such as IP addresses have been removed from the dataset to protect user privacy.

Results

As indicated in Table I, Minrva for iOS Downloads totaled 771, whereas Minrva for Android totaled 499 during the study. Total module hits recorded during this time period was found to be 45,974 with September being the highest month of use for the application, followed by October as the second most popular month for the app.

In Table II, it can be seen that the most popularly used parts of the Minrva mobile app during the first semester includes the catalog search module, technology module and the display module. Initial indications suggest that students in their first semester are interested in searching items within the library catalog. Students are also interested in learning more about what technologies are available in the library. Table III shows a breakdown by month of the most popular modules. I can see that catalog search was the most popular use within the mobile app for all months of the study. Note also that according to data from Table III the course reserves module was a highly used component of the app during November, which is toward the end of the semester.

Modules that were not heavily used included blocks, events and question board. Implications of students not making use of these modules are discussed in more details during the discussion and conclusion section that follows.

The author notes a limitation to an API-specific analysis: due to the methodology of server analysis-specific paths of user interactions from within the mobile app was not possible. Google website analytics offers a feature where the paths of users through a website are reported through an administrator panel. Future research studies could look at paths of engagement rather than sum API usage.

As a further limitation, it is possible that returning student or staff users may be represented in the use data. It is reasonable to assume, however, that a large number of observed downloads were likely from students who attended new student orientation because the downloads coincided with the months and weeks of summer new student orientation.

Discussion and conclusion

The catalog search module was found to be the most popular use for the Minrva app. Of those users who chose to download the app in the first semester of study, they searched the catalog more so than any other library use. I believe that such high use of searching is indicative of the strength of the library brand being "books". This also relates to more recent studies of learning analytics within academic research libraries. In a focused study on undergraduate student success, researchers have found a positive correlation with library checkouts noting:

students who borrowed books (including traditional books, e-books, and inter-library loans) were more likely than their peers who did not borrow books to be engaged in their academic work, to develop academic skills like critical thinking and reading/writing, and to engage in scholarly activities including analyzing materials, using facts and examples to support claims, and incorporate ideas and topics across disciplines (Soria et al. , 2017, p. 20).

While the catalog search module showed high uses in this study, the second-level module that a user is directed into after searching, the display module, was not correspondingly selected. This seems to indicate that while students searched and browsed the results from within the search interface, those users did not tap for more details on the items, which came up after they searched. It is not known at this time if this is because the results were not satisfactory or if the students were simply interested in browsing results lists of library items to get a sense of what is available in the collection. Mobile application use typically is a short interval of time, so this result may be an artifact of the mobile platform or it could be a larger design issue to address in the Minrva platform.

Designers of the Minrva mobile app thought that the display module could be useful as a springboard for saving a history of items people looked at and may want to reference again. However, designers may in fact have to integrate the "history" feature directly within modules rather than expecting users to select an item of interest. System designers should also take further time to investigate what low use means for a module by a first-year student. Is the placement of the module in the home screen a disadvantage - or is it the case that several of the least used apps simply do not have relevance to the first-year student's first semester experience? Without a corresponding qualitative analysis, such a finding would be only speculative at this time. With the modular options increasing over time, the Minrva platform is considering ways in which only the most used modules are initially presented to users and making available the less used modules within an "other" tab of the app. From a design perspective, it may offer a more streamlined experience and less cluttered look in the app.

As it relates to other popularly used portions of the app, I can see that the technology module was the second most popular module which students were interested in utilizing heavily. This finding is particularly value for academic libraries that loan out technology as part of their circulating collection because it underscores the interest and ongoing engagement which new student may have with such a resource. The technology collection is a growing and popular part of the undergraduate student landscape. A related finding from the undergraduate student design mobile app competition article indicated as much when students desire apps to provide information about the location they are currently studying in and what resources are available (Ward, Hahn, Mestre, 2015). In the Minrva app, the technology module is one of the custom modules in Minrva. It shows students a list of technologies that are available to check out from the location selected in the app. At this time, the custom module for technology exists in the undergraduate library location within Minrva. Noting the popularity and desire for up to the minute space information, the library has most recently made available a room reservation module from within the Minrva app so that students can book study rooms from their mobile device and also see what rooms are currently available in what campus library locations.

Future directions

While this study was focused on analyzing student interactions after the first downloads of the Minrva Library app, researchers want to undertake an analysis of those who continually use the app. Specifically, researchers are interested in understanding the engagement of new students as they become seasoned library users. As an accompanying research question to investigate, what do researchers and expert users like faculty do within the Minrva app if they have downloaded the app for their research?

Next steps for research directions within Minrva log analysis are to look into the nature and content of the catalog and journal searches within the app. Finally, researchers have not yet constructed a picture of all module uses in the Minrva app. A future study will address all module API hits since 2013 to provide a global picture of how the app is being used over the years 2013-2016; this would help investigate several of the aforementioned questions related to longitudinal mobile app uses within library environments. Because the present study was purely quantitative there is a need for a mixed methods approach in the future that uses focus groups of new and returning students to more thoroughly understand the use and user preferences of Minrva modules.

Notes

1. The Minrva app download page can be accessed here: https://minrvaproject.org/download.php (accessed 16 December 2016).

2. For a list with examples of the Minrva APIs see: http://minrvaproject.org/services.php (accessed 16 December 2016).

3. Speech API: https://cloud.google.com/speech/ (accessed 16 December 2016).

4. Vision API: https://cloud.google.com/vision/ (accessed 16 December 2016).

5. https://firebase.google.com/docs/analytics/ (accessed 16 December 2016).

[Figures and tables omitted: See PDF]

References

References

Böhmer, M., Hecht, B., Schöning, J., Krüger, A. and Bauer, G., (2011), "Falling asleep with angry birds, facebook and kindle: a large scale study on mobile application usage", in Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, MobileHCI '11, ACM, New York, NY, pp. 47-56.

Brown, J.J. and Hernandez, C.L., (2010), "Technology in orientation", (Monograph no. 13 3rd ed., pp. 117-129), in Ward-Roof, J.A., (Ed.), Designing Successful Transitions: A Guide for Orienting Students to College, University of South Carolina, National Resource Center for the First-Year Experience and Students in Transition, Columbia, (accessed 16 December 2016).

Fitz-Walter, Z., Tjondronegoro, D. and Wyeth, P., (2011), "Orientation passport: using gamification to engage university students", Proceedings of the 23rd Australian Computer-Human Interaction Conference, OzCHI '11, in ACM, New York, NY, pp. 122-125.

Galvin, D. and Sun, M., (2011), "Using web services for a mobile OPAC", available at: http://journal.code4lib.org/articles/4810, Code4Lib Journal, no. 13, (accessed 21 September 2016).

Greenfield, G.M., Keup, J.R. and Gardner, J.N., (2013), Developing and Sustaining Successful First-Year Programs: A Guide for Practitioners, Jossey-Bass, San Francisco, CA.

Hahn, J. and Ryckman, N., (2012), "Modular mobile application design", available at: http://journal.code4lib.org/articles/7336, Code4Lib Journal, no. Vol. 18 (accessed 21 September 2016).

Hahn, J., Ryckman, B. and Lux, M., (2015), "Topic space: rapid prototyping a mobile augmented reality recommendation app", available at: http://journal.code4lib.org/articles/10881, Code4Lib Journal, no. 30, (accessed 21 September 2016).

Hahn, J., (2015), "The student/library computer science collaborative", Portal: Libraries and the Academy, Vol. 15 no. 2, pp. 287-298.

Hahn, J., (2016), API analysis of the Minrva mobile app (May 2015 - December 2015), University of Illinois, Urbana-Champaign, available at: https://doi.org/10.13012/B2IDB-5495131_V1, (accessed 19 December 2016).

Jansen, B.J., (2006), "Search log analysis: what it is, what's been done, how to do it", available at: http://dx.doi.org/10.1016/j.lisr.2006.06.005, Library & Information Science Research, Vol. 28 no. 3, pp. 407-432, (accessed 16 December 2016).

Johnson, L., Adams Becker, S., Cummins, M., Estrada, V., Freeman, A. and Hall, C., (2016), NMC Horizon Report: 2016 Higher Education Edition, The New Media Consortium, Austin, Texas.

Long, P. and Siemen, G., (2011), "Penetrating the fog: analytics in learning and education", available at: http://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education, (accessed 21 September 2016).

Lown, C., Sierra, T. and Boyer, J., (2013), "How users search the library from a single search box", College & Research Libraries, Vol. 74 no. 3, pp. 227-241.

Mikkelsen, S. and Davidson, S., (2011), "Inside the iPod, outside the classroom", Reference Services Review, Vol. 39 no. 1, pp. 66-80.

Mischo, W.H., Schlembach, M.C., Bishoff, J. and German, E.M., (2012), "User search activities within an academic library gateway: implications for web-scale discovery systems", available at: http://dx.doi.org/10.4018/978-1-4666-1821-3.ch010, in Popp, M.P. and Dallis, D., (Eds), Planning and Implementing Resource Discovery Tools in Academic Libraries, IGI Global, Hershey, PA, pp. 153-173, (accessed 16 December 2016).

Peters, T.A., (1993), "The history and development of transaction log analysis", available at: http://dx.doi.org/10.1108/eb047884, Library Hi Tech, Vol. 11 no. 2, pp. 41-66, (accessed 16 December 2016).

Rosen, C. and Shihab, E., (2015), "What are mobile developers asking about? A large scale study using stack overflow", Empirical Software Engineering, Vol. 21 no. 3, pp. 1192-1223, doi: 10.1007/s10664-015-9379-3.

Ryckman, B. and Ryckman, N., (2016), "Minrva | Catalog", available at: http://minrvaproject.org/catalog.php, Vol. 21, (accessed September 2016).

Soria, K.M., Fransen, J. and Nackerud, S., (2014), "Stacks, serials, search engines, and students' success: first-year undergraduate students' library use, academic achievement, and retention", The Journal of Academic Librarianship, Vol. 40 no. 1, pp. 84-91.

Soria, K.M., Fransen, J. and Nackerud, S., (2017), "Beyond books: the extended academic benefits of library use for first-year college students", available at: http://crl.acrl.org/content/early/2016/01/25/crl16-844.full.pdf, College & Research Library, pp. 1-27, (accessed 28 September 2016).

Upcraft, M.L., Gardner, J. and Barefoot, B., (2005), Challenging and Supporting the First-Year Student: A Handbook for Improving the First Year of College, Jossey-Bass, San Francisco, California.

Ward, D., Hahn, J. and Mestre, L., (2015), "Designing mobile technology to enhance library space use: findings from an undergraduate student competition", Journal of Learning Spaces, Vol. 4 no. 1, pp. 30-40.

Wong, S., (2012), "Which platform do our users prefer: website or mobile app?", Reference Services Review, Vol. 40 no. 1, pp. 103-115.

Appendix. Top three modules by month by total hits