Introduction
Medical students are often overwhelmed by the volume of information they are required to learn,1,2 and the problem of information overload in medical education has been a running theme for more than a century.3 Abraham Flexner attempted to tackle this problem in his landmark 1910 report,4 and throughout the 20th century, there were at least 24 additional major reports that called for further medical education reforms, many of which cited the challenges posed by increases in medical knowledge.5 Despite the improvements that Flexner's recommendations brought to bear, some have argued that little has changed in the post-Flexnerian era6 and that the problem has remained.7 There may be several factors contributing to this, including the manner in which information is traditionally disseminated to medical students.
Medical education evolved as an instructor-centered model during the last century.2,7 Students were uninvolved in shaping their educational experiences,1 and traditional teaching approaches emphasized passive learning in the form of faculty lectures and lacked interactivity, as seen with text-based materials.8,9
In recent years, computer-assisted instruction (CAI), the Internet, and Web-based learning (WBL) have helped alter the medical education landscape. CAI was first introduced to medical education in the early 1960s.10 Initially, there was great excitement, and institutions such as Ohio State University, Massachusetts General Hospital, and the University of Illinois created their own CAI systems and then shared them with other schools as part of a consortium established by the Lister Hill Center of the National Library of Medicine.11–13 Excitement gave way to skepticism, as barriers such as machine incompatibility, the prohibitive costs of computing, and a lack of access to computers prevented widespread adoption.9 With the advent of the personal computer in the mid-1980s and the rapid rise of the Internet, which allowed for the delivery of CAI via WBL,14 there has been a proliferation of WBL self-study resources.
This has empowered students to take greater control of their learning experiences15,16 to tackle the issue of information overload. Medical education has become more learner-centered.17,18 Students are able to learn actively at their own pace and to take advantage of rich, interactive, multimedia resources such as question banks, online videos, multimedia textbooks, and much more.19
The PLS is a multimedia WBL platform that uses audiovisual mnemonics in combination with text-based materials, self-assessment modalities, and an interactive user interface to teach medical sciences. It is designed to improve memory retention and exam performance in medical sciences by leveraging the benefits of mnemonic instruction. Mnemonics have been shown to not only increase memory retention but also to improve higher-order thinking.20 Achieving this enhanced "learning with understanding" is a major focus of efforts of the new science of learning because it allows students to better apply their knowledge in diverse problems and arenas.21 For further discussion of the benefits of mnemonic instruction please see Levin and Levin;20 Rummel, Levin and Woodward;22 and Scruggs et al.23
Studies have demonstrated that multimedia WBL in medical education can increase efficiency24 and learning8 when compared with traditional teaching approaches. More research, however, is required to explore whether multimedia WBL can improve memory retention, performance in tasks that require higher-order thinking, and student satisfaction through the use of audiovisual mnemonics in medical education.
Material and methods
Subjects
Two hundred fifty-seven first-year osteopathic medical students at Midwestern University Arizona College of Osteopathic Medicine in Glendale were recruited to participate in the study from February until March 2013. Eighty-eight volunteered, one of whom later withdrew after experimental session 1. This participant was not included in the statistical analysis. The study protocol was reviewed and approved by the Midwestern University Institutional Review Board, and informed consent was obtained from study subjects.
PLS audiovisual mnemonic WBL platform
The PLS concept was created and developed by two of the authors (RR and AY) to help medical students learn and memorize medical science materials in a more effective manner. At the core of the PLS (see Figure 1 and Supplementary materials) are audiovisual mnemonics; each one is intended to convey important attributes and information about a medical science concept or topic. The subject matter includes concepts/topics that are commonly tested in medical school, including those that frequently appear on the Comprehensive Osteopathic Medical Licensing Examination and the US Medical Licensing Examination.
Enlarge this image.Figure 1 Screenshot of the Picmonic® Learning System. Notes: This example topic is rhinovirus, which is portrayed by the rhino characters with virus-like legs. The Picmonic® Card includes major attributes of the topic, which are listed in the left pane. Each attribute is incorporated into the picture in a creative and memorable manner and is associated with salient information. For example, one attribute of rhinovirus is that it is a picornavirus, which is represented by the pickle corn character. A story sequence portrayed by the picture then binds the attributes of the topic together.
In the process of creating a Picmonic® Card, a concept/topic of interest is selected, and then the most important attributes are identified and encoded into memorable characters, using the keyword mnemonic strategy. For a discussion of the process of encoding information into this format, please see Levin and Levin.20 A linear narrative is created to connect the characters together, and this story is illustrated as a picture and recorded as an audio file. These files, along with text-based definitions for each of the attributes and a topic summary, are then inputted into the PLS. Students can access the PLS and their library of Picmonic® Cards via a Web site portal, where they can perform self-assessment by ranking their mastery of concepts/topics and identifying areas of weakness.
Experimental design
Six disease topics were identified as the subject matter for this study: Sturge-Weber syndrome, neurofibromatosis type 1, Ewing's sarcoma, giant cell tumor, osteosarcoma, and tuberous sclerosis. To minimize contamination, disease topics to which the first-year medical students had not yet been exposed in their medical education curriculum were selected.
Audiovisual mnemonics and text-based materials were developed for these disease topics and inputted into the PLS according to the processes discussed previously. Self-assessment modalities of the PLS were inactivated because experimental constraints, including limited time and single use of the PLS, precluded the subjects' ability to take advantage of the feature. A text-only version of the content was created by aggregating the same text that was inputted into the PLS, including the definitions and topic summaries. Key terms were highlighted to emphasize important information. The audiovisual mnemonic version of the content differed from the text-only version in that it contained a colored, audiovisual mnemonic representation of the text and interactive components, such as the ability to zoom in on a certain character/attribute in the picture and view the associated definition. It did not contain highlighted key terms, as in the text-only version.
Two parallel desktop computer applications with input and output functionality were developed to deliver the experiment contents that included tests, a distraction task, and a satisfaction survey, in addition to the study materials. These applications differed only in their provision of either audiovisual mnemonic study materials via the embedded PLS (application A) or text-based only study materials (application B). The applications were loaded onto computers in a testing center, half with application A and half with application B.
Using a computerized algorithm, study subjects were randomly assigned (1:1) into the PLS group (those using audiovisual mnemonics via the PLS; application A; n=46) and a control group (those using text-based materials of the same information; application B; n=42) and were directed to the appropriate computer terminal.
Session 1
Pretest
On initiation of the experiment, the subjects took a 10-minute baseline knowledge free-recall pretest before exposure to the study materials. In this pretest, subjects were given only the headings of the 6 disease topics (eg, "Sturge Weber syndrome") and instructed to list all medical science facts they knew about the topic.
Study period
Subjects were given a 5-minute tutorial, which allowed them to familiarize themselves with the type of study materials they would encounter. A 30-minute study period followed in which detailed information corresponding to the 6 disease topics that were previously introduced in the pretest was provided, the PLS group was given access to the audiovisual mnemonic version of the information, and the control group was given access to the text-based version. One subject from the control group withdrew from the study at this point (n=41).
Immediate tests
A 3-minute scrambled-word distraction task, analogous to that described by Rummel, Levin, and Woodward,22 was administered to reduce the effect of the subject's short-term memory on test performance. First, a 10-minute free-recall test was given that was identical in format to the baseline knowledge free-recall pretest, except that the disease topics were listed in a different order. Next, a 10-minute paired matching test was given in which subjects were instructed to match medical science terms with the appropriate disease topics.
Satisfaction survey
Subjects evaluated their satisfaction and engagement with the study material they were assigned to by answering 5-point Likert items. The survey contained a balance of both positive and negative questions to minimize potential acquiescence bias.
Session 2
One week delayed tests
One week after the 30-minute study period during session 1, subjects returned to the testing center and were given 10 minute free-recall and paired-matching tests exactly as in session 1 (items were listed in a different order). In addition, subjects were given a 21-minute multiple-choice test to evaluate their ability to integrate knowledge and perform tasks requiring higher-order thinking.
Session 3
One month delayed tests
One month after the 30-minute study period during session 1, subjects returned once again to the testing center. They were given 10-minute free-recall and paired-matching tests exactly as in sessions 1 and 2 (items were listed in a different order), and they were given a postexperiment exit survey to assess their general learning habits.
Scoring procedure
Scoring of the free-recall tests was based on the number of medical science facts correctly recalled about the disease topics. A master listing of possible correct answers was created before testing, and the subject's answers were matched against this list. Subject's answers that were not found on the master list but were deemed to be correct were subsequently added to this list. After all amendments to the master list were made, the maximum possible score was 57. Scoring of the paired-matching tests was based on the number of correctly matched medical science terms with the appropriate disease topics, with a maximum possible score of 20. Scoring of the multiple choice test was based on the number of correct answers, with a maximum possible score of 16. Each item on the satisfaction survey was scored from 1 to 5 in accordance with the 5-point Likert scale. The highest score on a positive question was 1, and the highest score on a negative question was 5.
Endpoints
The primary endpoints were the differences in performance between the PLS group and control group subjects on immediate, 1 week, and 1 month delayed free-recall and paired matching tests. These were selected as the primary endpoints because these tests were delivered at all 3 sessions, and thus were the most reliable markers for assessing subject's memory retention. Secondary endpoints were the difference in performance between the PLS group and control group subjects on a 1 week delayed multiple choice test and self-reported assessments of satisfaction and engagement with the study material.
Statistical analysis
The mean test score and standard deviation were calculated for each test taken by subjects in the PLS and control groups. The mean test scores for the control group were subtracted from the mean test scores for the PLS group, and two-tailed unpaired t-tests were conducted on these values according to a type 1 error probability of 0.01. The same process was conducted for all 5-point Likert items in the satisfaction survey Results
To ensure there were no significant differences between the PLS and control groups, an assessment of the subject's baseline knowledge of the 6 disease topics was obtained by administering a free-recall pretest. The results of the pretest scores were very similar, with the experimental group knowing a mean of 0.74 medical science facts and the control group knowing a mean of 0.78 medical science facts about the diseases. The difference in baseline knowledge between the two groups was not statistically significant.
At the delayed testing session, subjects were asked whether they had studied any of the disease topics outside of the designated 30-minute study period in session 1; none reported doing so. The subjects who had used the audiovisual mnemonic version of the study materials scored higher on every test administered throughout the 3 sessions (see Table 1). The percentage differences in scores on free-recall and paired matching tests also increased over time, with the greatest difference seen in session 3 on the 1 month delayed tests (see Table 2). The differences in mean scores between the experimental and control groups were all statistically significant at the level of P<0.001.
Enlarge this image.Table 2 Difference in mean scores between the Picmonic® Learning System and control groups Notes: aExperimental group mean−control group mean; b(experimental group mean−control group mean)/control group mean; cstatistically significant at the level P<0.001.
The results of the satisfaction survey demonstrated that subjects using the audiovisual mnemonic version of the study materials delivered via the PLS were more satisfied and engaged with the content than those using text-based-only study materials (see Table 3). On average, PLS group subjects were more likely to report their study materials as an efficient use of their time (mean difference, 2.35; P<0.001). Further, they claimed a greater ability to maintain focus (mean difference, 2.19; P<0.001). The differences in mean values on the 5-point Likert items between the PLS and control groups were all statistically significant at the level of P<0.001.
Enlarge this image.Table 3 Satisfaction survey Notes: aAll questions are on the 5-point Likert rating scale; bstatistically significant at the level P<0.001. Abbreviations: SD, standard deviation; PLS, Picmonic® Learning System.
Discussion
The results of this experiment indicate that compared with subjects who used the text-only version of the content, subjects who used the audiovisual mnemonic WBL version of the content retained medical science information about disease topics to a greater degree and for longer. They also outperformed their text-only counterparts on an application task (multiple choice test) that demanded higher-order thinking skills than those called for in measures of memory requiring only rote memorization. Of great significance is the higher reported satisfaction and engagement with the audiovisual mnemonic WBL content. Subjects felt it allowed them to remember complex information to a greater degree and that it was a more memorable way to present the information. Fostering excitement with medical education content is an important goal any educator should strive for and must be a focus of learner-centered curricula. These results support previous research into the use of multimedia medical education programs8 and the use of mnemonic strategies for improving memory retention and the application of knowledge.22
The audiovisual mnemonic WBL version of the content delivered via the PLS has various components that do not exist in the text-only version, including audiovisual mnemonics and interactive graphical simulation. Given that audiovisual mnemonics are at the core of this multimedia program, the greatest benefits in memory retention and applicability of knowledge can likely be attributed to this aspect of the PLS. However, further research is required to determine how much each aspect of the PLS contributes to improved memory retention and applicability of knowledge.
The results of this study have several limitations. The study was conducted at only a single osteopathic medical school, so the findings must be established at other additional institutions to make them more generalizable. Although subjects using the audiovisual mnemonic WBL version of the content applied their information superiorly on a 1 week delayed multiple choice test, this test was not delivered immediately after the 30-minute study period, nor was it delivered after 1 month, as the free-recall and paired matching tests were. Thus, the change over time in ability to apply knowledge was unable to be assessed. Further, subject's performance on standardized medical exams, such as the Comprehensive Osteopathic Medical Licensing Examination and US Medical Licensing Examination, was not assessed. These are tests in which applicability of knowledge has real implications for medical students' careers. Traditional teaching approaches often include text-based materials and faculty-delivered lectures among other modalities; this experiment's text-only version of the content did not include these other modalities. Inclusion of lectures, for example, in addition to text-based materials would allow for a better comparison with broader traditional teaching approaches, not just traditional text-based materials.
Medical education curricula include many different types of content areas and learning objectives25 that may demand varying instructional strategies.26 More research is required to determine exactly what aspects of medical education should be taught using audiovisual mnemonic WBL versus other modalities, including traditional teaching approaches. Finally, this research examines knowledge/performance; however, as Cook27 has emphasized, this is only a substitute for the most important outcome, which is effect on physician performance and patient outcomes.28–30 Further study should attempt to tackle such questions.
Conclusion
This single-center pilot study suggests that audiovisual mnemonic WBL provides marked improvement compared with traditional text-based study materials for teaching the main attributes of disease topics. Audiovisual mnemonics delivered via the PLS can play a role in continuing the learner-centered revolution in medical education and this study supports their use to supplement and enhance traditional text-based instruction.
Acknowledgments
The authors thank Yue Ma, Hannah Dineen, and Dr Omar Badri for writing the test questions. We also thank Midwestern University for allowing use of their computer testing center.
Disclosure
AY, HG, MB, RR, JL, and SI have a financial interest in Picmonic®, LLC. The authors report no other conflicts of interest in this work.
Supplementary materials
Video 1 Walkthrough of a Picmonic® sample card. Available from: https://www.youtube.com/watch?v=gJoU-uy9nkY.
[RAW_REF_TEXT] 1. D’Eon MF, Harris C. If students are not customers, what are they? Acad Med. 2000;75(12):1173–1177. [/RAW_REF_TEXT] [RAW_REF_TEXT] 2. Gunderman RB. Achieving Excellence in Medical Education. 2nd ed. London: Springer; 2011. [/RAW_REF_TEXT] [RAW_REF_TEXT] 3. Small PA Jr, Stevens CB, Duerson MC. Issues in medical education: basic problems and potential solutions. Acad Med. 1993; 68(Suppl 10):S89–S98. [/RAW_REF_TEXT] [RAW_REF_TEXT] 4. Flexner A. Medical Education in the United States and Canada: A Report to the Carnegie Foundation for the Advancement of Teaching. New York: Carnegie Foundation for the Advancement of Teaching; 1910. [/RAW_REF_TEXT] [RAW_REF_TEXT] 5. Christakis NA. The similarity and frequency of proposals to reform US medical education. Constant concerns. JAMA. 1995;274(9):706–711. [/RAW_REF_TEXT] [RAW_REF_TEXT] 6. Fox RC. Time to heal medical education? Acad Med. 1999;74(10):1072–1075. [/RAW_REF_TEXT] [RAW_REF_TEXT] 7. Irby DM, Cooke M, O’Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med. 2010;85(2):220–227. [/RAW_REF_TEXT] [RAW_REF_TEXT] 8. Grundman JA, Wigton RS, Nickol D. A controlled trial of an interactive, web-based virtual reality program for teaching physical diagnosis skills to medical students. Acad Med. 2000;75(Suppl 10):S47–S49. [/RAW_REF_TEXT] [RAW_REF_TEXT] 9. Piemme TE. Computer-assisted learning and evaluation in medicine. JAMA. 1988;260(3):367–372. [/RAW_REF_TEXT] [RAW_REF_TEXT]10. Coleman K, Ehrlich L. Emerging technologies: Learning from history. Proc Annu Symp Comput Appl Med Care. 1984:975–979. [/RAW_REF_TEXT] [RAW_REF_TEXT]11. Weinberg AD. CAI at the Ohio State University College of Medicine (1973). Comput Biol Med. 1973;3(3):299–305. [/RAW_REF_TEXT] [RAW_REF_TEXT]12. Barnett G, Hoffer E, Famiglietti K. Computers in medical education: Present and future. Proc Annu Symp Comput Appl Med Care. 1983:11–13. [/RAW_REF_TEXT] [RAW_REF_TEXT]13. Harless WG, Drennon GG, Marxer JJ, Root JA, Miller GE. CASE: a Computer-Aided Simulation of the Clinical Encounter. J Med Educ. 1971;46(5):443–448. [/RAW_REF_TEXT] [RAW_REF_TEXT]14. Friedman RB. Top ten reasons the World Wide Web may fail to change medical education. Acad Med. 1996;71(9):979–981. [/RAW_REF_TEXT] [RAW_REF_TEXT]15. Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19(6):698–707. [/RAW_REF_TEXT] [RAW_REF_TEXT]16. Kamin C, Souza KH, Heestand D, Moses A, O’Sullivan P. Educational technology infrastructure and services in North American medical schools. Acad Med. 2006;81(7):632–637. [/RAW_REF_TEXT] [RAW_REF_TEXT]17. Zucker S, White JA, Fabri PJ, Khonsari LS. Instructional intranets in graduate medical education. Acad Med. 1998;73(10):1072–1075. [/RAW_REF_TEXT] [RAW_REF_TEXT]18. Cook DA, Dupras DM. Flexible teaching for inflexible schedules: an online resident curriculum in acute ambulatory care. Med Teach. 2003;25(3):330–331. [/RAW_REF_TEXT] [RAW_REF_TEXT]19. Santer DM, Michaelsen VE, Erkonen WE, et al. A comparison of educational interventions. Multimedia textbook, standard lecture, and printed textbook. Arch Pediatr Adolesc Med. 1995;149(3):297–302. [/RAW_REF_TEXT] [RAW_REF_TEXT]20. Levin ME, Levin JR. Scientific mnemonomies: methods for maximizing more than memory. Am Educ Res J. 1990;27(2):301–321. [/RAW_REF_TEXT] [RAW_REF_TEXT]21. Committee on Developments in the Science of Learning with additional material from the Committee on Learning Research and Educational Practice; Board on Behavioral, Cognitive, and Sensory Sciences; Division of Behavioral and Social Sciences and Education; National Research Council. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, DC: National Academies Press; 2000. [/RAW_REF_TEXT] [RAW_REF_TEXT]22. Rummel N, Levin JR, Woodward MM. Do pictorial mnemonic text-learning aids give students something worth writing about? J Educ Psychol. 2003;95(2):327–334. [/RAW_REF_TEXT] [RAW_REF_TEXT]23. Scruggs TE, Mastropieri MA, Berkeley SL, Marshak L. Mnemonic strategies: evidence-based practice and practice-based evidence. Intervention School Clinic. 2010;46(2):79–86. [/RAW_REF_TEXT] [RAW_REF_TEXT]24. Lyon HC Jr, Healy JC, Bell JR, et al. PlanAlyzer, an interactive computer-assisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease. Acad Med. 1992;67(12):821–828. [/RAW_REF_TEXT] [RAW_REF_TEXT]25. Learning objectives for medical student education – guidelines for medical schools: report I of the Medical School Objectives Project. Acad Med. 1999;74(1):13–18. [/RAW_REF_TEXT] [RAW_REF_TEXT]26. Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? A systematic review of the variability of interventions. Med Educ. 2010;44(8):765–774. [/RAW_REF_TEXT] [RAW_REF_TEXT]27. Cook DA. The research we still are not doing: an agenda for the study of computer-based learning. Acad Med. 2005;80(6):541–548. [/RAW_REF_TEXT] [RAW_REF_TEXT]28. Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79(10):955–960. [/RAW_REF_TEXT] [RAW_REF_TEXT]29. Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35(4):331–336. [/RAW_REF_TEXT] [RAW_REF_TEXT]30. Whitcomb ME. Research in medical education: what do we know about the link between what doctors are taught and what they do? Acad Med. 2002;77(11):1067–1068. [/RAW_REF_TEXT] Supplementary materials Video 1 Walkthrough of a Picmonic® sample card. Available from: https://www.youtube.com/watch?v=gJoU-uy9nkY.
1. D’Eon MF, Harris C. If students are not customers, what are they? Acad Med. 2000;75(12):1173–1177.
2. Gunderman RB. Achieving Excellence in Medical Education. 2nd ed. London: Springer; 2011.
3. Small PA Jr, Stevens CB, Duerson MC. Issues in medical education: basic problems and potential solutions. Acad Med. 1993; 68(Suppl 10):S89–S98.
4. Flexner A. Medical Education in the United States and Canada: A Report to the Carnegie Foundation for the Advancement of Teaching. New York: Carnegie Foundation for the Advancement of Teaching; 1910.
5. Christakis NA. The similarity and frequency of proposals to reform US medical education. Constant concerns. JAMA. 1995;274(9):706–711.
6. Fox RC. Time to heal medical education? Acad Med. 1999;74(10):1072–1075.
7. Irby DM, Cooke M, O’Brien BC. Calls for reform of medical education by the Carnegie Foundation for the Advancement of Teaching: 1910 and 2010. Acad Med. 2010;85(2):220–227.
8. Grundman JA, Wigton RS, Nickol D. A controlled trial of an interactive, web-based virtual reality program for teaching physical diagnosis skills to medical students. Acad Med. 2000;75(Suppl 10):S47–S49.
9. Piemme TE. Computer-assisted learning and evaluation in medicine. JAMA. 1988;260(3):367–372.
10. Coleman K, Ehrlich L. Emerging technologies: Learning from history. Proc Annu Symp Comput Appl Med Care. 1984:975–979.
11. Weinberg AD. CAI at the Ohio State University College of Medicine (1973). Comput Biol Med. 1973;3(3):299–305.
12. Barnett G, Hoffer E, Famiglietti K. Computers in medical education: Present and future. Proc Annu Symp Comput Appl Med Care. 1983:11–13.
13. Harless WG, Drennon GG, Marxer JJ, Root JA, Miller GE. CASE: a Computer-Aided Simulation of the Clinical Encounter. J Med Educ. 1971;46(5):443–448.
14. Friedman RB. Top ten reasons the World Wide Web may fail to change medical education. Acad Med. 1996;71(9):979–981.
15. Cook DA, Dupras DM. A practical guide to developing effective web-based learning. J Gen Intern Med. 2004;19(6):698–707.
16. Kamin C, Souza KH, Heestand D, Moses A, O’Sullivan P. Educational technology infrastructure and services in North American medical schools. Acad Med. 2006;81(7):632–637.
17. Zucker S, White JA, Fabri PJ, Khonsari LS. Instructional intranets in graduate medical education. Acad Med. 1998;73(10):1072–1075.
18. Cook DA, Dupras DM. Flexible teaching for inflexible schedules: an online resident curriculum in acute ambulatory care. Med Teach. 2003;25(3):330–331.
19. Santer DM, Michaelsen VE, Erkonen WE, et al. A comparison of educational interventions. Multimedia textbook, standard lecture, and printed textbook. Arch Pediatr Adolesc Med. 1995;149(3):297–302.
20. Levin ME, Levin JR. Scientific mnemonomies: methods for maximizing more than memory. Am Educ Res J. 1990;27(2):301–321.
21. Committee on Developments in the Science of Learning with additional material from the Committee on Learning Research and Educational Practice; Board on Behavioral, Cognitive, and Sensory Sciences; Division of Behavioral and Social Sciences and Education; National Research Council. How People Learn: Brain, Mind, Experience, and School: Expanded Edition. Washington, DC: National Academies Press; 2000.
22. Rummel N, Levin JR, Woodward MM. Do pictorial mnemonic text-learning aids give students something worth writing about? J Educ Psychol. 2003;95(2):327–334.
23. Scruggs TE, Mastropieri MA, Berkeley SL, Marshak L. Mnemonic strategies: evidence-based practice and practice-based evidence. Intervention School Clinic. 2010;46(2):79–86.
24. Lyon HC Jr, Healy JC, Bell JR, et al. PlanAlyzer, an interactive computer-assisted program to teach clinical problem solving in diagnosing anemia and coronary artery disease. Acad Med. 1992;67(12):821–828.
25. Learning objectives for medical student education – guidelines for medical schools: report I of the Medical School Objectives Project. Acad Med. 1999;74(1):13–18.
26. Cook DA, Garside S, Levinson AJ, Dupras DM, Montori VM. What do we mean by web-based learning? A systematic review of the variability of interventions. Med Educ. 2010;44(8):765–774.
27. Cook DA. The research we still are not doing: an agenda for the study of computer-based learning. Acad Med. 2005;80(6):541–548.
28. Chen FM, Bauchner H, Burstin H. A call for outcomes research in medical education. Acad Med. 2004;79(10):955–960.
29. Prystowsky JB, Bordage G. An outcomes research perspective on medical education: the predominance of trainee assessment and satisfaction. Med Educ. 2001;35(4):331–336.
30. Whitcomb ME. Research in medical education: what do we know about the link between what doctors are taught and what they do? Acad Med. 2002;77(11):1067–1068.
Adeel Yang,1,* Hersh Goel,1,* Matthew Bryan,2 Ron Robertson,1 Jane Lim,1 Shehran Islam,1 Mark R Speicher2
1College of Medicine, The University of Arizona, Tucson, AZ, USA; 2Arizona College of Osteopathic Medicine, Midwestern University, Glendale, AZ, USA
*These authors contributed equally to this work
© 2014. This work is licensed under https://creativecommons.org/licenses/by-nc/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.