Purpose: This study describes the system architecture and user acceptance of a suite of programs that deliver information about newly updated library resources to clinicians' personal digital assistants (PDAs).
Description: Participants received headlines delivered to their PDAs alerting them to new books, National Guideline Clearinghouse guidelines, Cochrane Reviews, and National Institutes of Health (NIH) Clinical Alerts, as well as updated content in UpToDate, Harrison's Online, Scientific American Medicine, and Clinical Evidence. Participants could request additional information for any of the headlines, and the information was delivered via email during their next synchronization. Participants completed a survey at the conclusion of the study to gauge their opinions about the service.
Results/Outcome: Of the 816 headlines delivered to the 16 study participants' PDAs during the project, Scientific American Medicine generated the highest proportion of headline requests at 35%. Most users of the PDA Alerts software reported that they learned about new medical developments sooner than they otherwise would have, and half reported that they learned about developments that they would not have heard about at all. While some users liked the PDA platform for receiving headlines, it seemed that a Web database that allowed tailored searches and alerts could be configured to satisfy both PDA-oriented and email-oriented users.
INTRODUCTION
This project uses personal digital assistants (PDAs) as conduits to promote and leverage electronic resources to library users. A suite of free, open-source applications has been developed to alert library users to new or updated content in electronic resources and newly acquired clinical books. This paper describes the system architecture and components and reports the results of a pilot project by a small group of library users.
The PDA Alerts program is a suite of applications designed for clinicians with Palm operating system (OS) PDAs. It consists of a small PDA application, a synchronization application residing on the user's personal computer, a librarian-in-the-loop interface application, and a relational database. Information coming to the health sciences library in electronic format is automatically sent to the project email account, which is read by the Librarian Interface application. New books and other information not received electronically are manually entered into the Librarian Interface application.
After review by a librarian, the information is coded and sent to the relational database. The clinician-users install a software package that creates a conduit from their personal computer to the relational database, using the Palm hotsync application with version 4.1 of the Palm Desktop. After installing the software and synchronizing their PDAs, clinicians can view headlines describing content newly added from a variety of library resources. At any given time, the fifty most recent headlines appear on the users' PDAs. Users mark headlines of interest in the PDA application using a checkbox form and, on their next PDA synchronization, receive links to selected resources in their email accounts. The source code and database schema [1] are available to other institutions through a Creative Commons Attribution-Noncommercial-Share Alike license [2].
The librarian selects the headlines to be included as alerts. During the pilot project, headlines were delivered from a variety of sources, including:
* Harrison's Principles of Internal Medicine [3]
* National Guideline Clearinghouse [4]
* Scientific American Medicine [5]
* Cochrane Database of Systematic Reviews [6]
* UpToDate [7]
* BMJ Clinical Evidence [8]
* National Institutes of Health (NTH) Clinical Alerts [9]
* New clinical books added to the library's collection
When the software was introduced, a research project was undertaken with the following specific aims:
* to determine the feasibility of providing the PDA Alerts service
* to learn which resources were the most requested by users of the service
* to see if the PDA platform conveyed any advantages over simply delivering these alerts via email
* to determine if usage of this service affected the frequency of visits to the library
BACKGROUND
A recent systematic review of handheld computing in medicine [10] notes that, while handheld computers have become increasingly prevalent in health care settings, many clinicians remain unaware of potential uses. Various types of alerting services have been developed for medical PDA users. Tables of contents and abstracts, sometimes including the full text of brief news items, are available for some journals through Highwire [11] and AvantGo [12-14]. These services are noninteractive, in that they do not allow users to request delivery of full-text articles corresponding to selected abstracts.
Other services allow PDA users to request delivery of additional information after viewing abstracted information on their handheld device. Users of Unbound Medicine's CogniQ platform [15] have access to Harrison's On Hand, BMJ Clinical Evidence, and Ovid@Hand and may request additional information be sent to a personalized Web page. Other PDA information sources send additional information to the user's email account upon request; this is the case with ePocrates's DocAlerts [16, 17], as well as with the PDA Alerts program described here.
A PDA-based approach to delivering headlines has five advantages over existing methods.
* First, it aggregates new content from a variety of disparate sources. Instead of each user having to subscribe to email updates from Harrison's, National Guideline Clearinghouse, Scientific American Medicine, UpToDate, and the library new book list and receiving separate emails from each resource, the PDA Alerts program allows the users to receive content alerts from a variety of resources in one convenient format.
* second, headlines are included from some resources where email updates are not readily available, including BMJ Clinical Evidence, the Cochrane Database of Systematic Reviews, and NIH Clinical Alerts.
* Third, users can filter the headlines by broad subject area. Instead of having to scan several separate email updates for items related to a given specialty, such as pediatrics, users can filter the headlines on their PDA to view only items related to their specialty.
* Fourth, unlike email, the PDA Alerts program can be used from anywhere: during a commute, while standing in line at the cafeteria, while awaiting lab results, or at any other time the PDA is at hand. The user selects headlines of interest, and additional information on each is sent via email at the next synchronization.
* Fifth, programs like PDA Alerts make it possible for clinicians to keep abreast of updates as they occur. The time may soon arrive when continuous updates might render the concept of numbered textbook editions obsolete. For example, a new edition of Harrison's Principles of Internal Medicine had been published about every three years. Now, portions are added and updated daily on Harrison's Online.
METHODS
The system architecture (Figure 1) consists of an alerts database and three individual modules. These modules are the Librarian Interface, the PC Synchronization Application (PC-Sync Conduit), and the Palm application. Users can download a list of headlines into their PDA, browse the list, and select items of interest. Synchronizing their PDAs sends their requests to the database, generates an email with the headlines, and retrieves a new list of available knowledge resources.
Librarian interface
The Librarian Interface (LI) is an application that resides on the librarian's computer. It opens the project mail account and validates messages by looking for specific words and phrases in the subject and sender fields. Some emails have hypertext markup language (HTML) message bodies containing the alerts information. The LI "reads" these emails and separates out the individual headline texts and corresponding uniform resource locators (URLs) from the message body and then puts them in a list for the librarian to review. Other emails contain a link to the alerts information. These Web pages are manually opened by the librarian and saved to disk. This source is then opened and read by the LI, and the individual headlines and URLs are entered into the application queue. The librarian opens the LI and then opens each email or Web page (Figure 2). Each email alert may have eight to ten distinct headlines.
After the librarian assigns subject categories to each headline to facilitate sorting on the handheld device, the LI sends them to the database for storage. Most of the emails and Web pages require only minimal editing by the librarian. In most cases, they take only five to ten minutes to edit, depending upon the number of links. The LI can be used to parse almost any Web page, and virtually anything with a URL can be added as a headline.
Alerts database
The alerts database is a simple relational database that stores headlines and user synchronization records. The headline table assigns a unique identifier (ID) to each headline and stores the URL, the headline, an assigned category, the source, and the input date. The headline table is updated whenever the librarian interacts with the LI application. Each time users synchronize their PDAs, a record is created storing the date and time and the users' email addresses. Each headline request by a user is assigned a unique ID, and a record is created storing the user's email, headline ID, request date, and the request ID.
PC synchronization application
The PC-Sync Conduit is a Java application that resides on the user's computer. It is activated when users synchronize their PDAs (using the hotSync application). PC-Sync sends any requests from the PDA to the alerts database and retrieves the matching URLs and headlines from the database. An email message with the requested headlines and URLs is sent to the user's email address. If a requested resource is Internet protocol (IP)-restricted but allows proxy access, a string is appended to the URL to authenticate the user. If the request is for a book, PC-Sync sends an email request to the librarian, who processes the request. Next, the PC-Sync removes all the headlines stored in the user's PDA and replaces them with the fifty most recent headlines from the alerts database.
Palm application
The Palm application is a very small application residing on the user's PDA, using less than 7K of memory. The majority of processing and storage occurs in the database or on the user's computer. The Palm application has been tested on PaImOS versions 3.5, 3.51, 4.0, and 4.1 using the Palm Emulator.
When opened, the Palm application checks the PDA for a valid email address. If an email address is not present, the user is prompted to enter one. An internal validation rule only accepts an email address from the institutional domain. The Palm application next displays the date of the last synchronization, the user's email address, and a list of source abbreviations. The alerts are displayed in a continuous scrollable format with the most recent headlines listed first. Each headline is numbered, has a selection box, and displays the source abbreviation and headline text. Clicking the headline shows a detail box with the date of entry, source, and category. Users can select one or more headlines with their PDA styluses. If they wish, users can filter headlines by subject category to view only headlines related to a given specialty such as pediatrics or women's health.
Usage study
After the study was approved by the institutional review board (IRB), volunteers were recruited via emails directed to clinical faculty in medicine, nursing, and allied health; to medical and doctoral nursing students; and to medical residents at an academic health sciences center. Electronic recruitment methods were used to target study volunteers who were familiar with technology. An email was sent to approximately 1,200 users describing the project and soliciting volunteers who had a PDA with a Palm OS that was synchronized with a desktop computer at least once a week. The email included a link to a pre-experiment survey. All recipients of the email were encouraged to complete the survey, even if they were not PDA users or did not wish to participate in the PDA Alerts project. One hundred fifty-seven completed surveys were received, after discarding duplicate, incomplete, and test surveys. Of the respondents to the survey, 81 (52%) stated they would like to participate; 28 (18%) stated they would not wish to participate; and 48 (30%) did not specify.
A second notice was sent to all survey respondents, except for those 28 individuals who had stated that they did not wish to participate. From this second emailing to the remaining 129 respondents, 25 respondents confirmed they wished to participate, and an additional 19 declined.
Of the twenty-five who confirmed a desire to participate, sixteen actually participated in the pilot project, including eight faculty members from the school of medicine, three resident physicians, four advanced practice nurses (including two nursing faculty, one nurse practitioner, and one research nurse), and a human resource specialist. Volunteers represented the fields of anesthesiology, health management and informatics, internal medicine, family medicine, nursing, obstetrics/gynecology, and surgery.
Enlarge this image.Table 1Proportion of headlines posted versus headlines requested from each resource
During the research period of approximately four months, data were gathered on the frequency of PDA synchronizations for each participant and the number of headlines requested. The specific headlines requested were also logged and tallied for each of the resources. Participants completed a post-experiment survey at the conclusion of the study to gauge their opinions about the service. A separate follow-up was done with those participants who had not requested any headlines to identify any barriers encountered.
RESULTS
Over the course of the project, 816 headlines were delivered to users' PDAs generating a total of 131 requests from the 16 users (Table 1). Scientific American Medicine generated the highest proportion of headline requests at 35%, followed by UpToDate with 28%. Twenty-five percent of the headlines from both BMJ Clinical Evidence and Harrison's Online were requested, followed by 18% from the Cochrane Database of Systematic Reviews and 15% from the National Guideline Clearinghouse. The proportion of new clinical books requested was somewhat smaller. While new clinical books added to the library collection accounted for the majority of headlines added, only 11% of the 373 book alerts were requested.
A chi square test revealed that the difference in request patterns for new clinical books versus the other types of resources was significant at the 0.01 level. Two NIH Clinical Alerts were released during the study period, but neither of those generated any requests from alerts users. Because some resources contributed many more headlines than other resources did, an attempt was made to normalize the data by calculating the proportion of headlines requested versus headlines contributed for each resource. As shown in Table 1, twice as many headlines were requested from Harrison's as from Scientific American Medicine (28 versus 14), but the proportion of headlines requested from Scientific American Medicine (35%) was 10% higher than for Harrison's (25%).
Usage patterns among study participants varied widely. Five participants (31%) requested an average of at least one headline per synchronization; on the other hand, two participants (13%) requested no headlines at all during the study period. A closer look at the five heaviest users revealed that three were advanced practice nurses, one was a faculty member in obstetrics/gynecology, and one was a member of the anesthesiology faculty.
A separate follow-up was conducted near the end of the study for four participants who had downloaded the application but never requested any headlines to learn if this was due to having hardware or software problems, forgetting to check the alerts headlines on their PDAs, or not finding any headlines of interest to them. Two of these individuals subsequently requested some headlines and consequently received a post-experiment survey along with the other participants. One of the remaining two non-requesters was lost to follow-up; the other non-requestor replied that email notifications would have been more effective for him:
I find myself too busy to look at the headlines as there is a never ending flow of info coming my way. I don't spend downtime with my palm, which is probably why this hasn't been a useful model for me. On the other hand, I do read a tremendous amount of emails, including several table of contents services. Perhaps in my case, that form of notification would be more effective.
Two study participants who had requested headlines declined to complete the full post-experiment survey, because they felt that they had not used the application enough to be able to fully evaluate it. Just as for the non-requester quoted above, lack of time was a major obstacle to use for these two participants, one of whom commented:
I just did not have the time to look at the alerts like I thought I would. I did on one occasion and remember finding something of interest, but was never able to follow through. Just plain too busy.
The post-study survey form was sent to twelve participants and completed by eight of them. Four participants never responded to any of the three requests to complete the post-experiment survey. This amounted to a response rate of 67% on the post survey (n = 12). Although detailed statistics would not be particularly meaningful given the small number of respondents, their responses helped the study team form some overall impressions of the acceptance of the project.
Half of the participants who completed the postexperiment survey deemed the PDA Alerts application somewhat useful to extremely useful for keeping up to date. The respondents felt the PDA Alerts were more useful to them in their clinical encounters and in keeping current but less helpful in a teaching or research context. Most responded that they learned about new medical developments sooner than they otherwise would have; in fact, half of the respondents indicated that they learned about new medical developments through the PDA Alerts that they otherwise would not have learned about at all.
However, respondents were divided on their feelings about the appropriateness of the handheld platform for this type of alerting service. While three participants (38%) liked receiving alerts via the PDA, the other five respondents to the post-experiment survey did not feel that their handheld computer worked well for this type of service. Of those five, three would have would have preferred to receive their alerts via email, echoing the comment of the non-requester reported above. Seven of the eight respondents to the post-experiment survey reported no change in the number of visits to the health sciences library. Only one reported that using the PDA Alerts software reduced his number of trips to the library. When asked what other types of alerts, resources, or headlines they would like to have delivered to their PDAs, users mentioned "drug information," "evidence-based guidelines," and "more public health and nursing" headlines.
DISCUSSION
Six of eight respondents to the post-experiment survey indicated that they would like to be able to search the alerts database on the Web in the place of or in addition to viewing the fifty most recent headlines on their PDAs. That way, users could create their own on-demand alerts to view headlines from a given resource, from a specific range of dates, from a specific subject category (e.g., pediatrics, surgery), or from some combination thereof. A Web-accessible database that would be searchable on demand would also address the comment from one user that the headlines were "too general" and that the list of fifty headlines was "too many to scroll through." And perhaps, for those four individuals who indicated they would have preferred email notifications for this type of service, a selective dissemination of information (SDT) profile could be created on such a Web database, so that they could receive email notifications whenever new headlines of interest to them were added.
The unobtrusiveness of the PDA Alerts program turned out to be something of a mixed blessing. Once the program was installed, users had to remember to open the application on their PDAs to view new headlines as they arrived. To remind users of the study and encourage usage, emails were sent to participants every one to two weeks during the study period. Despite these emails, one participant did not request any headlines until very late in the study period, simply because she did not know where to find the headlines on her PDA. Another user cited the same problem as an obstacle to use on the post-experiment survey. This misunderstanding occurred despite frequent emails to users throughout the study period, all of which included a telephone number and an email address to contact if they had questions or encountered problems.
Another alerting program, ePocrates DocAlerts [18] promotes usage by opening automatically whenever the PDA is started to reveal any new, unread headlines retrieved in a previous synchronization as a way to promote and encourage use. Yet informal feedback from some of the users in this study indicated that they would uninstall any software that took such an intrusive approach. Perhaps the ideal solution would be to make this a configurable option in the software, giving each user the opportunity to specify "unobtrusive" or "obtrusive" mode for displaying new PDA Alerts.
Some participants' preference for email as a means of delivering the alerts might reflect the level to which the PDA had been integrated (or not integrated) into their daily work. Everyone used email; not everyone spent their "downtime" with their PDAs.
One surprising development during the course of the study was how infrequently some of the participants synchronized their PDAs with their desktop computers. It was anticipated that most PDA users would synchronize with a desktop computer at least a few times per week. Indeed, of those who indicated an initial interest in participating on the pre-experiment survey and who responded to the question about synchronization frequency, 81.5% stated they synchronized at least once per week (n = 76). In reality, the average time between synchronizations for the study participants was 14.7 days, ranging from a low of 1.89 days to a high of 47.5 days between synchronizations.
None of the participants reported any major bugs or software problems during the course of the project. However, three of the respondents to the post-experiment survey noted that it took too long to synchronize their PDAs. This was probably due to the infrequency of their synchronizations rather than the performance of the PDA Alerts software.
There is no way of knowing for certain how to account for the low rate of participation. Software installation was required, and, although the entire installation took less than five minutes, it did involve multiple steps, including two synchronization operations. Members of the project team also offered "house calls" to assist with installation of the software. Still, it would have been easier to encourage participation if the application could simply have been beamed to users.
Modifying behavior so that clinicians integrated usage of the program into their daily routines was also problematic. This integration could have been helped by making the alerts pop to front whenever the handheld was turned on or by using the PDA Alerts software to deliver announcements from department chairs or other types of material that would be perceived as "required reading." However, given these participants' preference for email as a delivery mechanism, maybe there is little justification for further attempts to modify behavior to promote usage of handheld computers.
The pervasive theme in partipants' comments was lack of time: lack of time to install the software, lack of time to view the headlines, and lack of time to follow-up on headlines of interest. Previous studies reveal that most questions that arise in patient encounters go unanswered [19, 2O]. If clinicians do not even have the time to hunt for information in response to questions directly arising from patient encounters [21, 22], perhaps it is unrealistic to think they would have the time to forage for new information [23, 24], even when it is delivered to their handheld devices.
CONCLUSIONS
Most users of the PDA Alerts software reported that they learned about new medical developments sooner than they would have otherwise, and half of the respondents reported that they learned about developments that they would not have heard about at all. The PDA service delivered alerts from disparate sources in one format, including alerts to new content in BMJ Clinical Evidence and the Cochrane Database of Systematic Reviews, for which alerts were not readily available through any other means. While some users liked the PDA platform for receiving headlines, it seemed that a Web-accessible database that allowed tailored searches by date, resource, and subject featuring an SDI alerting capability could be configured to satisfy both the PDAoriented and email-oriented users.
Technical problems with the software were few and minor, and the program was sufficiently flexible to support delivery of virtually any kind of text or headline and then to email corresponding URLs in response to the users' selections. The source code and database schema are available to other institutions through a Creative Commons Attribution-Noncommercial-Share Alike license.
ACKNOWLEDGMENTS
The authors thank the other members of the MIAIMS Alerts PDA Project team for their assistance: WiI Reeves, Chetan Soni, Deborah H. Ward, and Devang Thacker.
* Funded in part by grant DHHS G08 LM05415-07 from the National Library of Medicine.
[dagger] Based on a paper presented at MLA '03, the 103rd Annual Meeting of the Medical Library Association; San Diego, CA; May 6, 2003.
REFERENCES
1. Source Code for MIAIMS PDA Alerts, [cited 12 Nov 2003]. < http://www.muhealth.org/~library/pdapages/ sourcecode.html >.
2. CREATIVE COMMONS. Creative Commons licenses. [Web document]. Creative Commons, 2003. [cited 17 JuI 2003], <http : / / creativecommons.org / licenses />.
3. HARRISON'S ONLINE. Email alert sign up. [Web document]. McGraw-Hill, 2001-2003. [cited 14 JuI 2003]. <http:// harrisons. accessmedicine.com / server-j ava / Arknoid / amed / harrisons / su_whats_new /emaiLalert_subscribe.html >.
4. NATIONAL GUIDELINE CLEARINGHOUSE. NGC update service: subscription management. [Web document]. Washington, DC: National Guideline Clearinghouse, 1998-2003. [cited 14 JuI 2003]. <http://www.ngc.gov/whatsnew/register .aspx>.
5. SCIENTIFIC AMERICAN MEDICINE. What's new in ACP medicine. [Web Document]. WebMD. [cited 14 JuI 2003]. <http:// pref.health.Webmd.com/sam/SAMSelection_miniform.asp>.
6. OVID TECHNOLOGIES. Evidence based medicine reviews: Cochrane database of systematic reviews. [Web document]. Ovid Technologies, 2003. [cited 14 JuI 2003]. <http://www .ovid.com/site/catalog/DataBase/904.jsp? top = 2&mid=3 &bottom=7&subsection=10>.
7. UpToDate. [Web document]. UpToDate, 2003. [cited 14 JuI 2003]. <http://www.uptodate.com>.
8. BMJ PUBLISHING GROUP. Clinical evidence. [Web document], BMJ Publishing Group, 2003. [cited 17 JuI 2003]. <http://www.clinicalevidence.com>.
9. US NATIONAL LIBRARY OF MEDICINE. NATIONAL INSTITUTES OF HEALTH. NIH clinical alerts and advisories. [Web document]. Bethesda, MD: National Library of Medicine, 2003. [cited 14 JuI 2003], <http://www.nlm.nih.gov/databases/alerts/>.
10. FISCHER S, STEWART TE, MEHTA S, WAX R, LAPINSKY SE. Handheld computing in medicine. J Am Med Inform Assoc 2003 Mar-Apr;10(2):139-49.
11. HIGHWIRE PRESS. My PDA channels. [Web document]. Board of Trustees, Stanford University, 2001-2003. [cited 14 JuI 2003]. <http://highwire.stanford.edu/cgi/alerts/>.
12. AVANTGO. Wiley InterScience MobileEdition. [Web document]. AvantGo, 1998-2003. [cited 14 JuI 2003]. <https:// my.avantgo.com/browse/se0 arch/0/3287/>.
13. AVANTGO. AFP Abstracts by HandheldMed. [Web document]. AvantGo, 1998-2003. [cited 14 JuI 2003]. <https:// my.avantgo.com/browse/search/O/1482/>.
14. AvANTGo. BMC Research. [Web document]. AvantGo, 1998-2003. [cited 14 JuI 2003]. <https://my.avantgo.com/ browse/se0 arch/0/3178/>. (link no longer active, 22 Jun 2004.)
15. UNBOUND MEDICINE. Solutions: CogniQ. [Web document]. Unbound Medicine, 2000-2003. [cited 14 JuI 2003]. <http://www.unboundmedicine.com/cogniq.htm>.
16. ROTHSCHILD JM, LEE TH, BAE T, BATES DW. Clinician use of a palmtop drug reference guide. J Am Med Inform Assoc 2002 May-Jun;9(3):223-9.
17. EPocRATES. DocAlert messages. [Web document], ePocrates, 2003. [cited 14 JuI 2003], <https://www.epocrates .com/docAlerts.do>.
18. IBID.
19. ELY JW, OSHEROEF JA, EBELL MH, CHAMBLISS ML, VINSON DC, STEVERMER JJ, PIEER EA. Obstacles to answering doctors' questions about patient care with evidence: qualitative study. BMJ 2002 Mar 23;324(7339):710.
20. GORMAN PN, HELFAND M. Information seeking in primary care: how physicians choose which clinical questions to pursue and which to leave unanswered. Med Decision Making 1995 Apr-Jun;15(2):113-9.
21. DORSCH JL. Information needs of rural health professionals: a review of the literature. Bull Med Libr Assoc 2000 Oct;88(4):346-54.
22. COVELL DG, UMAN GC, MANNING PR. Information needs in office practice: are they being met? Ann Intern Med 1985 Oct;103(4):596-9.
23. SHAUGHNESSY AF, SLAWSON DC, BENNETT JH. Becoming an information master: a guidebook to the medical information jungle. J Fam Pract 1994 Nov;39(5):489-99.
24. NUTTING PA. Tools for survival in the information jungle. J Fam Pract 1999 May;48(5):339-341.
Received November 2003; accepted April 2004
By E. Diane Johnson, MES, AHIP
Head, Information Services
J. Otto Lottes Health Sciences Library
University of Missouri-Columbia
Columbia, Missouri 65212
Paul E. Pancoast, MO
Post Doctoral Fellow
Joyce A. Mitchell, PhD
Professor of Child Health
Department of Health Management and Informatics
University of Missouri-Columbia
Columbia, Missouri 65211
Chi-Ren Shyu, PhD
Assistant Professor
Department of Computer Engineering and Computer Science
University of Missouri-Columbia
Columbia, Missouri 65211
Copyright Medical Library Association Oct 2004