Question: What are the challenges involved in designing, modifying, and improving a major health information portal that serves over sixty million page views a month?
Setting: MedlinePlus, the National Library of Medicine's (NLM's) consumer health Website, is examined.
Method: Challenges are presented as six "studies," which describe selected design issues and how NLM staff resolved them.
Main Result: Improving MedlinePlus is an iterative process. Changes in the public user interface are ongoing, reflecting Web design trends, usability testing recommendations, user survey results, new technical requirements, and the need to grow the site in an orderly way.
Conclusion: Testing and analysis should accompany Website design modifications. New technologies may enhance a site but also introduce problems. Further modifications to MedlinePlus will be informed by the experiences described here.
INTRODUCTION
MedlinePlus [I], the National Library of Medicine's (NLM) health information Website, provides the public with authoritative health information. Using a set of selection guidelines [2], MedlinePlus organizes links to information from the National Institutes of Health (NIH) and other government agencies and links to information from carefully evaluated professional and voluntary health organizations. These links are organized on over 700 "health topic pages" and are supplemented by licensed content including drug information, a medical encyclopedia, a dictionary, news feeds, and tutorials. MedlinePlus builds on the strengths of NLM and NIH. The library has a long and rich history of selecting and organizing health information and is the creator of MEDLINE/PubMed [3], the world's premier source of medical research information.
MedlinePlus is designed and built by NLM staff, who work with medical librarians from around the country to maintain the site [4]. Content reviewers and selectors use a Web-based database input system to create, review, and maintain site records, links to health information on other Websites. Site records are used to build pages of links, called health topic pages. The input system also allows selectors to perform comprehensive reviews of pages on an ongoing basis. Selectors work on a development Website, and reviewers then publish the revisions to the MedlinePlus production database. In addition to these functions, the MedlinePlus team maintains static hypertext markup language (HTML) pages (e.g., selection guidelines, FAQs), featured sites, and spotlights. NLM information technology staff maintains the input system that synchronizes information in the development and production areas [5].
The MedlinePlus public user interface brings together the wealth of information from health topic pages, features, and licensed content. MedlinePlus integrates these resources to provide one-stop shopping for consumer health information. Changes in the public user interface are ongoing, reflecting trends in Web design, recommendations from usability testing, results of user surveys, technical requirements of new content, and the need to grow the site in an orderly way.
The purpose of this paper is to inform health information professionals who use MedlinePlus about the challenges of designing, modifying, and improving the site. The authors highlight six issues involved in maintaining a major health information portal that now serves over sixty million page views a month. These challenges are presented as "studies," or vignettes, which describe selected design issues and the ways NLM staff resolved them. The paper concludes with a look at future MedlinePlus design challenges and opportunities.
THE HOME PAGE AND NAVIGATION STRUCTURE
Page design-including graphics, markup, and usability-has been an ongoing challenge for the MedlinePlus team. The site has grown tremendously over the past six years, and pages need to reflect this growth in an orderly way. The home page needs to be dynamic, drawing users back regularly and providing visitors with fresh material. Interior pages must reflect a common navigation structure and maintain site branding. Extensive usability testing, customer comments, Web log analysis, user surveys, and focus groups inform all changes to the site's design [6].
Home page design
The MedlinePlus home page that debuted in November 2002 was designed to be simple, intuitive, and accessible to users with virtually any level of Web expertise (Figure 1). Pages have a uniform look and feel. Branding is simple with a uniform, distinctive color palette. The first column of the home page contains the primary site navigation with seven distinct categories. These categories appear as tabs on secondary-and tertiary-level pages. The center column features changing and fresh content. Health news, updated daily, appears at the top and includes a link to a news page with headlines by date. Featured Site and In the Spotlight highlight new Websites and health topics. The third column contains a link to the multimedia tutorials, which are quite popular among MedlinePlus users. Additional links include ClinicalTrials.gov [7] and NIHSeniorHealth.gov [8], other NLM consumer sites. The second and third columns are designed to be flexible; content may be changed as needed to respond to NLM priorities for disseminating health information.
Designing for Web standards
Many Web standards and usability issues influence the technical design of MedlinePlus. All pages are generated using the HTML 4.01 transitional document type definition (DTD) standard. Cascading style sheets (CSS) version 1 are used to generate text attributes and some positioning. NLM information technology staff plans to convert the site to fully comply with extensible HTML (XHMTL) 1.0 and CSS version 2.
All MedlinePlus pages have moving elements because of the site's liquid design (the width of the page can expand or shrink to any size browser window). Staff has found it difficult to implement a CSS-based, table-free design that both maintains this fluid display and works gracefully on older browsers. Using full CSS has many advantages, including the reduction of page file size by as much as 20% to 25% through the elimination of unnecessary tables and other markup. For accessibility reasons [9], sizing of text must be controlled by users. All of these technical requirements, from HTML to CSS, are balanced with usability priorities.
Navigation structure
Users can find information in a variety of ways. Alphabetic A-Z lists and a simple, sitewide search box at the top of every page are the primary methods of navigation (Figure 2). A-Z lists enable simple drill-down navigation and are available on category "home pages" such as Health Topics, Drug Information, and Medical Encyclopedia. The lists are highly visible, table-formatted text links. A-Z sub-navigation is also available on health topic, drug, and encyclopedia pages. Usability findings consistently showed that users navigated effectively using the A-Z selection lists [10].
HEALTH TOPIC PAGES
Organization and integration
Health topic pages are the core of MedlinePlus and function as portals to a wide variety of carefully selected information produced by NIH and other organizations. Each page consists of links to full-text Web documents organized into categories such as overviews, treatment, and prevention. On every health topic page, a table of contents lists the categories and helps consumers easily jump to a specific section. Each health topic page also displays related topics and categories for additional site navigation. For example, the Breast Cancer page links to the related health topics, Mastectomy and Mammography, and the topic group, Women's Health.
Pages also contain links to other related NLM products and services, including preformulated MEDLINE/PubMed searches, clinical research studies currently recruiting participants from ClinicalTrials.gov, and genetic information from Genetics Home Reference [H]. NLM's Medical Subject Headings (MeSH) [12] is the mechanism used to link many of these products and services to each health topic. The NIH institute with primary responsibility for the topic is also listed. Additional features include licensed news stories and local health services. This extensive integration of a variety of information is what makes MedlinePlus unique among consumer health information Websites.
Design
Since the birth of MedlinePlus, achieving the optimal design of the health topic pages has been challenging, especially the placement of the table of contents and links to related subject content. One of the first designs for topic pages positioned the table of contents at the top of the page and featured the MEDLINE/PubMed searches in a colored box to the side (Figure 3). The page displayed related topic links directly under the page title, and a bar separated the table of contents from the content links. Usability testing identified a number of problems with this design. Instead of scanning the table of contents or exploring the page, testers often clicked on the related topic links and looped through the site. This was particularly true if the vocabulary of the related topic matched the user's vocabulary. Too much white space around the table of contents pushed the linked content further down the page. The horizontal bar under the table of contents acted as a "scroll stopper," discouraging exploration of the page. The preformulated MEDLINE/PubMed searches were so visible that users were instantly drawn to them, although they should be secondary for consumers seeking health information [13].
NLM modified this design based on extensive usability testing and changing expectations in Web design. The redesign in 2000 eliminated the left sidebar on all pages. This change allowed additional space for more relevant content at the top of health topic pages. The table of contents was moved to the left side and grouped into subcategories. Content links displayed at the top of the page. The primary NIH institute, if one existed for the topic, was placed below the table of contents, followed by the MEDLINE/PubMed searches and related topics. This design was similar to the link-rich portals that were popular on the Web in 2000. Yahoo! and other sites featuring categories with many links were heavily used and often required fewer than three clicks to reach the content. Web designers were creating flat sites with content positioned close to the top of pages.
This health topic page design served MedlinePlus users well for a few years. However, as NLM continues to expand the number of health topics and add new features to health topic pages, the potential for user confusion has increased. Each new feature appears in a colored box in a third column on the right side of the page (Figure 4). Recent usability testing indicated that many consumers ignored these colored boxes because they resembled Web advertisements. They also failed to notice the table of contents on the left side of the page. Testers preferred to drill down to the information and did not mind clicking more than three times, as long as the "scent," or trigger words and careful layout, were present [14]. Other findings included the desire to see overviews and introductory material first. These findings will contribute to a new look for the health topic pages.
Linking to off-site content
MedlinePlus links to full-text resources from NIH, other government agencies, and selected organizations. Leading users to these resources without taking them away from MedlinePlus is an ongoing dilemma. The first MedlinePlus design presented users with an intermediate page stating, "You are leaving MedlinePlus." Initial usability testing showed this "boundary page" was an annoyance and suggested that clicking should spawn a second, smaller window. After implementing the spawned window, NLM received fewer complaints, although some Web users continued to have usability problems (such as when spawned windows open behind the MedlinePlus browser or open as a minimized icon). The advent of ad blockers now poses new barriers to this design, and it is likely to undergo further evaluation.
IMPLEMENTING A DATABASE-DRIVEN WEBSITE
MedlinePlus was built with a database back-end and Web-enabled application very early in its development. Information collected for each Website, such as page titles and document descriptions, needs to be consistent across all pages, and selectors and reviewers need global methods for editing topic page information. Storing site records in a database ensures that they are unique and consistent across multiple topic pages. The application provides a forms-based system to control the data and workflow and enables rapid development cycles [15]. The public Website does not access the database directly; an HTML, "static" version of the site is built nightly from the database, allowing NLM to use its existing Web management tools including Web statistics software [16].
Page display
One of the challenges of designing a database-driven site is the difference between the ideal display of content and the automatic generation of page displays. Health topic pages need to display a wide range of data types and data lengths, and the design must take into account variations between the longest and shortest strings, the inclusion and exclusion of data elements, and so on, because the database outputs this information according to a prescribed template. For example, some health topic pages contain MEDLINE/PubMed searches, while others do not. The placement of the colored box containing the MEDLINE/PubMed searches must appear integral to the page design if present but not leave an obvious gap if absent (Figure 4).
Translating HTML design mockups into database-generated code presents other concerns. Design elements may need receding to accommodate the embedding of database queries or to prevent the insertion of extra lines and spaces. For example, the "featured site" module allows the MedlinePlus team to create and store customized features in the database. Features can be copied and modified for repeated use. A "preview" module allows feature creators to view a potential feature before it becomes public. Occasionally, the featured site generated by the database does not match the HTML mockup. In these cases, NLM staff can accept the database-generated version or manually add HTML code to the input form for the featured site. This manual addition of HTML code adds time but provides more flexibility for the final design.
Database design
Using a database can involve a tradeoff between quick product releases and a loss in design functionality. Early decisions about the database structure and rules have required modifications as MedlinePlus has matured. MedlinePlus's initial development entailed a series of quick product releases, but its growth has necessitated a variety of changes that revisited previous decisions. For example, one early database decision that required modification was allowing only one "see" reference per topic on health topic index pages. Thus, the reference for Manic-Depressive Illness could point to only one topic, Depression. Usability research showed that cross-references improve site navigation [17] (e.g., Manic-Depressive Illness see Bipolar Disorder; Depression), but the database permitted only a "one-to-one" relationship, not a "one-to-many" relationship that was necessary to make this display possible. In 2001, a redesign of the input system implemented the "one-to-many" relationship. A parallel database implementation allowed each news story posted to MedlinePlus to link to one or more health topics, news from the same date, and news on the same topic. The database adds critical flexibility and linkages that greatly enhance the MedlinePlus design.
Related programming rules
Another early decision was to create uniform resource locators (URLs) for MedlinePlus health topic pages automatically by deriving them from health topic names. This ensured consistency, but it also meant that if the name of the topic were changed, the new topic name triggered the creation of a new URL, breaking bookmarked links to the original URL. For example, changing the topic name "Cancer (General)" to "Cancer," would change the topic's URL from http://www.nlm .nih.gov/medlineplus/cancergeneral.html to http:// www.nlm.nih.gov/medlineplus/cancer.html. This constrained staff from changing the names of topics even when topic names became outmoded (removing the word "General" from topics) or new topic names made older names obsolete (changing "Impotence" to "Erectile Dysfunction"). Staff reprogrammed URL creation in 2003 to allow needed flexibility, creating a mapping table that facilitated automatic redirection from old URLs.
ADDING LICENSED CONTENT
MedlinePlus health topic pages provide a complete guide on a disease or condition. However, topic coverage depends on other organizations to publish electronic documents that meet the MedlinePlus selection guidelines. Some health topics are not well covered on the Web by organizations meeting the guidelines, so NLM supplements the site with licensed content.
MedlinePlus licenses a medical encyclopedia, drug information, health news, a medical dictionary, and interactive health tutorials. It is the responsibility of the content providers to keep this material current. Content providers either host the data themselves or provide it, in some specified format, to NLM. In the latter case, providers send regular updates to NLM. A seamless interface between licensed content and the existing MedlinePlus health topic pages, a usable navigation system for licensed content, and the ability to update content in a timely manner are necessary to provide the best user experience.
When NLM first licensed the A.D.A.M. [18] Medical Encyclopedia in 1999, adam.com hosted the files, and NLM provided the company with the MedlinePlus header and footer templates. Adam.com "wrapped" the content with these templates so that the navigation and look and feel were consistent with the rest of MedlinePlus. It was time consuming for NLM staff and adam.com to coordinate the updates to header and footer templates. In addition, if the adam.com server had service outages, NLM had little control over when the problem would be fixed.
Further, NLM could not modify the interface of the encyclopedia content. Adam.com organized the encyclopedia by broad categories: Diseases, Injury, Nutrition, Poison, Special, Surgery, Symptoms, and Tests. Though each of these categories contained an A-Z navigation list, all seven broad categories appeared at the top of every encyclopedia page. Usability testing revealed confusion about the meaning of the category links. NLM resolved these issues when adam.com delivered the data as formatted HTML files to be hosted on the MedlinePlus Website, permitting modification of the interface. Now, one unified encyclopedia "home page" with an A-Z navigation list contains all the articles, replacing the broad categories. Accompanying illustrations now appear at the top of the encyclopedia pages and internal navigation of the multiple image presentations has been improved. Subsequent usability testing has validated the efficacy of these design choices.
When NLM licensed the United States Pharmacopeia [19] for drug information monographs, Micromedex [20], the distributor of this information, sent the library loosely formatted HTML files. These HTML files proved difficult to manage, but the problems were eventually solved with the delivery in a more standardized data format, extensible markup language (XML). Despite the complexity of providing and manipulating XML data, it proved to be a much more reliable format.
NLM does not edit the content of the drug monographs or encyclopedia files, so these files are never managed in a database. They reside as static files in the MedlinePlus file system and are formatted with style sheets. The style sheets apply the MedlinePlus template rules, such as font color, link color, list formatting, and so on.
In 2001, NLM began licensing news feed services. NLM selects relevant health-related news stories to include in MedlinePlus. This content-on-demand comes via an "editor's desk" middleware, which staff uses to select and download desired stories. Once downloaded, the stories reside in the MedlinePlus database for thirty days. As mentioned previously, the database enables each news story to link to multiple health topics and related news pages.
The Merriam-Webster Medical Dictionary, added to MedlinePlus in 2003, has presented few design challenges. The dictionary home page has a simple search box, and results open in a new window, serving up content hosted by Merriam-Webster. NLM has developed a co-branded logo, featuring both MedlinePlus and Merriam-Webster, for the dictionary results pages. This logo rarely needs updating, and these pages do not have MedlinePlus headers and footers. Unlike the initial experience with the medical encyclopedia content, consumers find the vendor-hosted dictionary design functions well.
The interactive tutorials, produced by the Patient Education Institute (PEI) [21] and built using Flash technology, are an invaluable source for patients, including those with low vision or low-literacy skills. The tutorials are mounted on NLM servers, and the site provides a Flash detection check. Because Flash files are self-contained, NLM does not make any changes to the tutorials. If changes are warranted, the library requests that PEI modify the content and templates.
IMPLEMENTING A SEARCH ENGINE
Implementing a search engine presented two design challenges: configuring the indexing engine to obtain optimal search results and displaying the search results in a meaningful way. Searching was not emphasized in the first MedlinePlus design, because the site's content was limited and users often received a message that their search found no results. Initially, MedlinePlus used ht://Dig [22], a freeware utility, which was configured to retrieve only pages on the MedlinePlus site. NLM wanted users to start on its own pages so they would see the variety of information and links related to a health topic.
In the next version of the site, an A-Z list of links at the top of the home page encouraged users to choose a topic by letter. NLM hoped to steer users to the list of existing links and minimize frustration from "no results" searches. The search link remained largely invisible to users, and NLM often received emails requesting the addition of a search feature. The world of Web searching was also changing. The emergence of Google had a major impact on user expectations. Users wanted a quick, simple way to search from every page. MedlinePlus had also changed, expanding the number of health topic pages and adding a variety of licensed content. In 2000, MedlinePlus introduced a search box on every page.
When planning for MedlinePlus en espanol, ht:// Dig was no longer adequate, because it could not accept input in Spanish, either with or without diacritics. The requirement for a new search engine allowed NLM to revisit the display of search results as well. Ht://Dig displayed a list of links regardless of content source, such as encyclopedia or drug information. This had certain advantages, such as retrieval of a drug record first after typing in the name of a drug, but also disadvantages, such as display of an encyclopedia entry ahead of a more comprehensive health topic page.
The replacement search engine, Recommind [23], not only searches diacritics, but also bases search results on an examination of the links appearing on the health topic pages. An undifferentiated display was no longer feasible, because it would interfile links on other Websites with MedlinePlus content. NLM adopted a file folder approach, dividing results into sections based on content areas of the site. The health topics section includes a link to the health topic as well as the content links found on that topic. Other folders contain results from the medical encyclopedia, drugs, and news (Figure 5).
However, recent usability testing has revealed some confusion with the folder display and search box domain on lower-level pages. In the medical encyclopedia section, some users expected to search only the medical encyclopedia pages. Users liked the folder display but often were not aware of its function. NLM is investigating other options, including flexible ranking and ordering schemes for results and better use of metadata in search refinement. This investigation may lead to a new search engine.
DESIGNING A BILINGUAL SITE
Since its release in the fall of 2002, MedlinePlus en espanol (Figure 6) has grown in size and usage, receiving over eleven million page views per month from over one million unique users from around the globe. Based on focus groups and early usability testing, NLM staff determined that the Spanish site's design, search, and navigation functions should match and integrate with the existing English structure. The goal was to design a site that Spanish speakers, English-speaking information intermediaries, health care professionals serving Hispanic patients, and public and hospital librarians could equally use.
Enlarge this image.Figure 5Search engine results, December 21, 2004Figure 6MedlinePlus en espanol home page, November 2, 2004
Language issues
With a well-tested English design already in place, the Spanish site could mimic most design elements, such as navigation buttons, tabs, headers, and footers. This worked well except when the Spanish word or phrase was considerably longer than its English equivalent. Buttons, tabs, and even headers behaved in unexpected ways. The lack of consistent Spanish Internet terminology compounded the problem. Short English names such as "FAQ," "home page," or "site map" had no equivalent in Spanish. The solution was a compromise between using alternate but still meaningful Spanish words and phrases and a slightly different English and Spanish display.
Focus group feedback about inconsistent navigation in Spanish on primarily English-language Websites determined the need for another language-based design decision. Spanish speakers expressed frustration with Websites that included links in Spanish, implying that the link would take them to more information in Spanish, only to find that the linked page was in English. All links to English information on MedlinePlus en español are clearly marked "ingles" to avoid confusion.
Spanish-speaking Internet users, like their Englishspeaking counterparts, rely on Web search functions to locate information quickly. But not all Spanish search terms are created equally. Spanish-speaking users may set their keyboards to "Spanish" and enter search terms with diacritics. Other users with standard US keyboard settings will not type terms with diacritics into a search box. Most search engines cannot handle words with diacritics and often treat them as two completely unrelated words or phrases. The MedlinePlus search engine indexes the Spanish pages both with and without diacritics. Users obtain identical results regardless of how they typed in their terms.
Navigation issues
Early in planning for a Spanish version, NLM staff recognized that there was significantly less health information in Spanish on the Web than in English. "Dead-end" navigation problems appeared as staff reviewed Spanish sites. The layout of a Spanish site would appear parallel to its English equivalent until users drilled down, only to find the equivalent Spanish information did not exist. Often users would find far less Spanish information or several equivalent English pages combined into a single information page in Spanish. Users would be left to wonder whether they navigated incorrectly, they had translated from one language to the other incorrectly, or the desired Spanish page just did not exist. To eliminate such dead-end navigation, MedlinePlus has a toggle, an "English" or "español" button, which takes users back and forth between each equivalent English and Spanish page.
The information architecture is the same for both MedlinePlus and MedlinePlus en espanol. Pages are organized using similar templates so content areas, topic subcategories, navigation buttons, and special features can be found in nearly the same place. The español/English toggle displays prominently in the upper right-hand corner of each MedlinePlus page. Implementing a consistently located, one-to-one toggle between topic pages, encyclopedia articles, and drug monographs has eliminated navigation complexity and language translation guesswork. Context-sensitive "Sorry" pages tell the user what information is not available and offer navigation to other parts of the site that may be useful.
To design the one-to-one English to Spanish mapping, NLM added a new field, the "mapped URL," to the input system. The mapped URL indicates identical page content in the other language. Populating this field in one language automatically populates the field in the other language. For example, on the Folk Acid page, the March of Dimes' Folic Acid FAQ includes the text, "Also available in Spanish." The word "Spanish" is hyperlinked directly to the Spanish-language document. An English-speaking intermediary does not need to navigate to the Spanish topic page to find information on Acido folico for their Spanish-speaking patron or patient. Librarians and health care professionals highly value this linking mechanism and can be confident that they are providing reliable health information to Spanish-speaking patients.
FUTURE CHALLENGES
The next major design challenge for the MedlinePlus team is providing access to information about local health services through a "Go Local" series of Websites. NLM has built a Web development environment for medical libraries and partnering organizations to organize local services linked to MedlinePlus health topics. NLM maintains the hardware and system software for hosting the records and producing the Go Local Web pages, as well as the vocabulary and mapping between the Go Local applications and MedlinePlus, the zip code look up and image maps, and a search engine for each locality. Participants connect through the Internet and log in to the system at NLM to create and edit records.
NLM controls the overall organization and design of the Go Local sites, although participants have customization options, such as inserting a logo. The goal is to create sites that are organized logically, have clear labels, include useful navigation, and comply with accessibility standards. Designs were usability tested, and the first sites launched in spring 2005.
Go Local presents numerous design issues. For example, conveying the difference between health information from MedlinePlus and health services from local areas is a major hurdle; users need a clear sense of who sponsors the Go Local site. Designing for how users expect to search geographically is another. In early usability testing, some participants wanted to use a city/town combination, while others expected the site to offer a zip code lookup. Because it offers a new kind of information, Go Local will continue to undergo extensive usability testing. In addition, popular search engines have started to offer searches of local services [24-26]. As Web users become accustomed to these searches, it will be important that Go Local monitor resulting research about user experiences. Go Local presents a complex set of issues in organizing and branding information.
MedlinePlus must also cope with managing the tremendous growth in excellent health information on the Web. The abundance of information on some of the more prevalent diseases and conditions necessitates the redesign of the health topic pages to balance the public's desire to see the most specific information that will answer their question with the danger of information overload on the pages.
Other concerns include better integration of licensed content products into health topic pages using metadata. A format for pushing information, really simple syndication (RSS), needs to be evaluated for its use in alerting subscribers to new health topics and news stories. MedlinePlus en espanol will undergo further testing to ensure the design meets the needs of Spanishspeaking information seekers. Finally, as more health information is translated into the many languages spoken in the United States, an accommodation for multiple languages requires attention.
CONCLUSION
Improving MedlinePlus is an iterative process. Testing and analysis accompany every design modification. There are few easy decisions; trade-offs between timely product releases and carefully tested designs are sometimes necessary and may need to be revisited if they do not work out as anticipated. As MedlinePlus adds more content and features, the public interface must change to accommodate them. Myriad factors influence the site's design, and there is tension between the need to keep up with the fast pace of the evolving expectations of Web design and the risk of confusing Website users. New technologies may enable a richer design but may also introduce usability and accessibility barriers. Web users' expectations change, and standard practices evolve at a rapid rate. The implications for MedlinePlus are clear: the design is always evolving and will continue to present ongoing challenges.
ACKNOWLEDGMENTS
The authors acknowledge Eve-Marie Lacroix, Joyce E. B. Backus, and Dianne Sun, of the National Library of Medicine, for their assistance with this article.
REFERENCES
1. NATIONAL LIBRARY OF MEDICINE. MedlinePlus. [Web document]. Bethesda, MD: The Library, [cited 2 Nov 2004]. <http: / / www.medlineplus.gov>.
2. NATIONAL LIBRARY OF MEDICINE. MedlinePlus: selection guidelines. [Web document]. Bethesda, MD: The Library, [cited 2 Nov 2004]. <http://www.rdm.nih.gov/medlineplus/ criteria.html>.
3. NATIONAL LIBRARY OF MEDICINE. MEDLINE/PubMed. [Web document]. Bethesda, MD: The Library, [cited 2 Nov 2004]. <http://www.ncbi.nlm.nih.gov/entrez/query.fcgi>.
4. MILLER N, LACROIX EM, BACKUS JEB. MEDLINEplus: building and maintaining the National Library of Medicine's consumer health Web service. Bull Med Libr Assoc 2000 Jan; 88(1):11-7. (Available from: <http://www.pubmedcentral. gov/articlerender.fcgi?artid=35193>. [cited 6 Sep 2005].)
5. MILLER N, TYLER RJ, BACKUS JEB. MedlinePlus: The National Library of Medicine brings quality information to health consumers. Libr Trends 2004 Fall;53(2):375-88.
6. WOOD FB, SIEGEL ER, LACROIX EM, LYON BJ, BENSON DA, CID V, FARISS SA. Practical approach to e-govemment Web evaluation. IT Professional 2003 May/Jun;5(3):22-e.
7. NATIONAL INSTITUTES OF HEALTH. Clinicaltrials.gov. [Web document]. Bethesda, MD: The Institutes, [cited 2 Nov 2004]. <http:/ /www.clinicaltrials.gov>.
8. NATIONAL INSTITUTES OF HEALTH. NIHSeniorHealth.gov. [Web document]. Bethesda, MD: The Institutes, [cited 2 Nov 2004]. <http://nihseniorhealth.gov>.
9. ARCHITECTURAL AND TRANSPORTATION BARRIERS COMPLIANCE BOARD. Electronic and information technology accessibility standards. [Web document]. Washington, DC: The Board, 2001 Feb 20. [cited 21 Dec 2004]. <http://www .access0 -board.gov/sec508/508standards.htm>.
10. NATIONAL LIBRARY OF MEDICINE. Evaluation of MedlinePlus and MedlinePlus en espanol: usability test results [internal document]. Bethesda, MD: American Institutes for Research, October 2003.
11. NATIONAL LIBRARY OF MEDICINE. Genetics home reference. [Web document]. Bethesda, MD: The Library, [cited 2 Nov 2004]. <http://ghr.nlm.nih.gov>.
12. NATIONAL LIBRARY OF MEDICINE. Medical Subject Headings (MeSH). [Web document]. Bethesda, MD: The Library, [cited 2 Nov 2004]. <http://www.nlm.nih.gov/mesh/ meshhome.html>.
13. MARILL JL. Designing a usable health information portal: the MedlinePlus experience from the National Library of Medicine, hi: Campbell N, ed. Usability assessment of library-related Websites: methods and case studies. Chicago, IL: American Library Association, 2001:100-8. (Library and Information Technology Association Guide, no.7.)
14. SPOOL J. Scent of a Web page or getting users to what they want. Presented to: National Cancer Institute. User Interface Engineering. 2002.
15. ANTELMAN K. Getting out of the HTML business: the database-driven Website solution. Inf Tech Libr 1999 Dec; 18(4):176-81.
16. MARILL J. So you say you want a redesign: usability, accessibility and the constraints of designing a database-driven Website. Presented at: Library and Information Technology Association National Forum, Nov 2000. (Available from: <http://www.nlm.nih.gov/pubs/staffpubs/lo/litaforum/ index.htm>. [cited 6 Sep 2005].)
17. KOYANI SJ, BAILEY RW, NALL JR. Research-based Web design and usability guidelines. [Web document]. Washington, DC: Department of Health and Human Services, 2003. [cited 10 Nov 2004]. <http://usability.gov/pdfs/guidelines .html>.
18. A.D.A.M. [Web document], [cited 2 Nov 2004]. <http:// www.adam.com>.
19. United States Pharmacopeia. [Web document], [cited 6 Nov 2004]. <http://www.usp.org>.
20. Thomson Micromedex. [Web document], [cited 6 Nov 2004]. <http://www.micromedex.com>.
21. Patient Education Institute. [Web document], [cited 2 Nov 2004]. <http: / /www.patient-education.com>.
22. Ht://Dig. WWW search engine software. [Web document]. The ht//Dig Group, 1995-2004. [cited 2 Nov 2004]. <http://www.htdig.org>.
23. Recommind. WWW search engine software. [Web document]. Recommind, 2004. [cited 2 Nov 2004]. <http:// www.recommind.com>.
24. STERLING G. Local search: the hybrid future. Search Engine Watch [serial online]. 8 Jan 2004. [cited 6 Nov 2004]. <http: / / searchenginewatch.com / searchday / article.php / 3296721 >.
25. SHERMAN C. Yahoo targets Google, Yellow Pages with new local search. Search Engine Watch [serial online]. 3 Aug 2004. [cited 6 Nov 2004]. <http://searchenginewatch.com/ searchday / ar ticle.php /3389591>.
26. PRICE G. Ask Jeeves goes local, adds new smart search features. Search Engine Watch [serial online]. 4 Aug 2004. [cited 6 Nov 2004]. <http://searchenginewatch.com/ searchday / article.php / 3390141 >.
Received January 2005; accepted June 2005
By Jennifer L. Marill, MA, MS, PMP*
Senior Systems Librarian
Naomi Miller, MLS
Manager, Consumer Health Information
Paula Kitendaugh, MSLS
Head, Health Information Products Unit
Public Services Division
National Library of Medicine
8600 Rockvilk Pike
Bethesda, Maryland 20894
* Currently, project manager, Office of Strategic Initiatives, Library of Congress, 101 Independence Avenue SE, Washington, DC 20540.
Copyright Medical Library Association Jan 2006