Continuing medical education and the anesthesiologist

Abstract

There are a large variety of scheduled activities and courses available to meet the continuing medical education (CME) needs of anesthesiologists. The presentation of CME material varies in format and delivery style. The reasons for attending CME activities include licensure requirements, participation in state and national societies, keeping current with technology, review of old subject material, participation as a lecturer, and other personal reasons. Funding occurs via personal funds, employer support, commercial support, or by research grants. External bodies, such as the American Council of Continuing Medical Education and the American Medical Association, have imposed guidelines in these areas. Methods to evaluate CME activities include retrospective needs analysis based on exit interviews, prospective needs assessment, focus groups, and complex systems such as the CRISIS criteria. Self-directed CME can be evaluated by data collection that identifies how quickly information is received and by the effect of this data on measurable outcome. In the future, CME will increasingly utilize simulators and multimedia computers. Multimedia can bring CME to the physician as opposed to the physician traveling to a CME site. Virtual reality and artificial intelligence are on the horizon and may interface well with the field of anesthesiology due to the technical nature of the discipline and the increasing use of computers and electronic data collection already occurring in clinical practice.

Introduction

The majority of states require physicians to attend and verify completion of continuing medical education (CME). Subsequently, a wide variety of postgraduate CME activities are available with different formats in varied locations for every subspecialty of organized medicine. Anesthesiology is no exception. The financial responsibility is often partially or completely assumed by the physician's employer, be it a hospital, foundation, or group. Those who absorb the cost themselves often designate this activity as a business expense, and the cost (at least in part) becomes a tax deduction. Society membership frequently entitles members to reduced tuition, but travel expenses remain.

Because there are so many ways to meet CME requirements and resources are considerable,¹ the question often asked by Anesthesiologists and their employers relates to quality and value of CME courses. This presentation attempts to address which CME activities within Anesthesiology are the most effective. The following topics are covered: 1) what is available, 2) how it is presented, 3) reasons for participation, 4) funding, 5) evaluation, and 6) future directions.

CME options for anesthesiologists

The American Society of Anesthesiologists (ASA) provides a biannual publication in January and July to all of its members, entitled "Calendar for Meetings," which is the most comprehensive listing of CME activities within Anesthesiology. The 1998 edition of Calendar for Meetings lists 250 different CME activities within the United States and Canada, an additional 40 to 50 abroad, and some with related topics of interest to anesthesiologists, such as Echocardiography, Advanced Neurological Monitoring, Internal Medicine, and Cardiology. Potentially, there are as many as 400 to 500 courses held annually, which would provide CME credits for an Anesthesiologist.

In addition to formal lectures, there are a variety of self-study programs that provide opportunity for CME credit. The best known of these programs is the ASA Self-Evaluation Examination (SEE), which is provided to members for a nominal fee. The program includes a biannual test to be taken by the participant, a critique of the questions, and a bibliography. If the test sheet is completed and returned to the SEE office, CME credit is awarded. The direction of the SEE task force, as determined by the ASA, is to service the educational needs of the Anesthesiologist in practice as opposed to the needs of anesthesia residents preparing for the American Board of Anesthesiology (ABA) In-Training or Written Board Examination.

The structure of CME activities vary widely. The highest profile are the 4- to 5-day national meetings of the ASA and the International Anesthesia Research Society (IARS). These offer the greatest number of contact hours per meeting and a wide variety of options, possible because of their size. There are also a large number of annual state society meetings, such as those of New York (NYPGA) and California, which attract large numbers of registrants, have large amounts of resources, and offer many courses. Smaller state societies typically offer 1- to 3-day courses with a smaller number of CME hours. Their appeal lies in their geographical proximity to their membership, scheduling on weekends when clinical activity is low, and their low tuition cost. Large city and county societies often provide similar activities on a yet smaller scale, usually for a single weekend day or weekday evening.

Subspecialization in Anesthesiology has proliferated, and societies, such as the Society for Education in Anesthesia (SEA), the American Society of Regional Anesthesia (ASRA), the Society for Ambulatory Anesthesia (SAMBA), the Society for Cardiovascular Anesthesia (SCA), and the Society of Pediatric Anesthesia (SPA), have been formed. These society meetings offer a forum for the gathering of individuals with similar subspecialized interests and allow a vehicle for the presentation of very focused research. They often offer technical refresher activities for the nonspecialized clinician who foresees an increasing need to provide services in those areas or wishes to attend just for interest.

Among the freestanding meetings aimed principally at postgraduate education, the Review and Refresher Courses predominate. Review courses aim to be comprehensive, lasting 4 to 5 days, and provide a high number of CME hours, but they also have a relatively high registration fee. These courses attract distinguished faculty and often are used by residents and those who have recently completed residency in preparation for the written portion of the Board Certification process. The new recertification examination for Anesthesiology has also stirred interest in the review courses for those planning to take the Continued Demonstration of Qualifications (CDQ) exam of the ABA. Refresher courses, on the other hand, attempt to cover small focused areas in Anesthesiology comprehensively. Single-subject didactic courses might pertain to a subspecialty, such as pediatric anesthesia, and workshops focus on areas like difficult airway management. Panels can include discussions of new drugs, products, or techniques.

Reasons for CME participation

State and licensing boards

In 1995, the majority of states required documentation of CME to renew medical and paramedical licenses. The majority of subspecialties (21/23 [91%]) accredited by the American Board of Medical Specialties have either recertification programs or a time-limited certification of credentials. ² State programs in Michigan,³ Connecticut,⁴ Minnesota,⁵ and Washington² specify requirements for the type and amount of credit hours. Some credits can be earned through self-instruction, although a percentage must be documented, are subject to audit, and have penalties associated with noncompliance.

Recertification

Recertification should stimulate modern practice and encourage ethical behavior.⁶ The ABA has recently adopted a policy for a 10-year duration of initial certification to take effect after the year 2000. This would complement the already existing CDQ program. Some mandatory recertification programs incorporate required CME.⁷ The quantification of physician CME in "units" was driven by the American Medical Association (AMA), which also developed the AMA/Physician's Recognition Award (AMA/PRA) and the American Council for Continuing Medical Education (ACCME). The ACCME defines and controls CME activities for the AMA/PRA. ⁸ Further development of self-directed learning and CME research are the future goals of the ACCME.⁹ Similar activities exist with the Royal College of Physicians and Surgeons of Canada¹⁰ and the American Osteopathic Association.¹¹ Subspecialty Recertification is described in detail for Obstetrics and Gynecology in Australia by Nambiar.¹²

Risk management and quality assurance

There has been some interest in linking CME with risk management.¹³ One example, not related to Anesthesiology, occurred in Massachusetts and focused on the issue of proper deep venous thrombosis (DVT) prophylaxis.¹⁴ A follow-up questionnaire designed to evaluate the DVT information program confirmed increased DVT prophylaxis in the practice of those who participated and presumably improved patient outcome, as DVT is the leading cause of inpatient morbidity. Almost two-thirds of 127 medical schools surveyed regarding CME activity responded that they had been driven primarily by risk management. ¹⁵ The rapid advancement of biomedical science also has contributed to the linkage of CME with quality assurance. New technologies are evolving from experimental to "standard of care" in progressively shorter intervals.^16,17 The concept of the need to keep current with new drugs, devices, and procedures is discussed by Curry and Wergin.⁶ Retraining individuals is as essential as retooling older manufacturing plants and must be done effectively and efficiently.

Remedial CME

In medicine, critical incidents, litigation, and peer review have created the need for remedial CME. Several states have formal programs, which are summarized in ^{Table 1} .^18-20 In the state of New York, about 200 physicians are censured annually and required to take remedial CME, programs that were developed by the Deans of the Associated Medical Schools of New York.¹⁸ Evaluation of remedial CME may be best modeled after undergraduate medical education.²⁰ Internationally, in New Zealand, an anesthetic mortality reporting system is the basis for the material selected by the Continuing Education Committee.²¹

There are other reasons for attending CME activities. Invited speakers attend to teach and lecture. Others attend to facilitate administrative work within the sponsoring organization. Preparation for primary and subspecialty certification (i.e., Pain, ICU) and CDQ constitute significant reasons for CME attendance.

CME attendance can also be favorably influenced by needs assessment surveys²² and by tailoring subsequent activities to those needs. A general interest in medicine as a whole and socioeconomic impact of disease will always be a part of the motivation for CME.²³ Because the practice of medicine involves interactions of human beings, CME can involve the teaching of communication and interpersonal skills.²⁴ A favorable image of the location on a CME brochure was found to be more effective in attracting registrants than a text cover.²⁵

Funding for CME

Because there is a cost associated with CME activity it is reasonable to ask, "Who pays for these costs?" Some CME is funded directly by the individual physician, although the time away from practice must be allowed by the hospital or employer.^26-28 In the past, however, much of continuing education was funded commercially, often by choosing selected speakers paid for directly by the sponsor. There is an increasing trend away from this kind of funding for CME because of the perceived potential for commercial support to influence scientific judgment about drugs or procedures.

Commercial funding of CME

The AMA and the Food and Drug Administration both require clear identification of any speaker or faculty member who has commercial interest in a subject, product, or procedure presented in a CME activity.²⁹ According to ACCME standards, this financial interest, as well as other conflicts of interest, must be defined in the syllabus material and verbally stated for oral presentation. Should a speaker fail to sign a conflict-of-interest release statement, the course director is obliged to disclose this fact in writing. A written conflict-of-interest policy must exist for an institution to accredit CME activities for AMA/PRA contact hours. ³⁰ Furthermore, commercial activity must not be concurrent with CME activity.³¹

In Italy, commercial funding is permitted, but funds are managed by a board completely independent of the CME activity.³² A survey in Germany³³ noted the potential for commercially sponsored CME to support a given company's product and approach to use of that product. There is a requirement that promotion of the product be distinctly separate from information about approaches to treatment utilizing that product. Physicians self-police this commercial support to avoid governmental control.

The alliance between CME and industry is not without benefit. The financial resources associated with product development allow for rapid dissemination of information. The vast majority of investigators who are called on by industry to give CME presentations about new drugs or devices present data, not sales information, in a CME setting. ^33,34 Peer review by course directors, as well as by educated participants, further increases the likelihood that the information presented is not biased. Voluntary industry standards encourage complete disclosure of the results of clinical trials at CME activities.

Employer payment

Employers fund much of the costs associated with CME for physicians. Anesthesiologists, being hospital based, usually work for a hospital or a group. Employers are induced to reimburse CME costs in this setting due to the CME requirements for licensure. Employers will demand the best value for their money, but an objective way to assess this will be difficult. Partial funding of CME from grants will become less common as the number of funded grants with this amount of latitude decreases. In countries, such as Great Britain or Scandinavia, where health care is funded by the government, CME also tends to be governmentally funded. It is provided at regional centers and is structured to fit the practice setting. ³⁵ Courses are scheduled throughout the year and on appropriately predetermined days of the week.³⁶ In Norway, governmentally funded committees create specialty-specific CME.³⁵ In underdeveloped countries, universities are presumed to be the providers of CME.³⁷ The burden on universities to be the providers of CME is not as great in the United States due to the strong commercial interests involved in the provision of CME.³⁸

Influence of health care reform

Health care reform in the United States will probably modify the structure of CME.³⁹ Emphasis on recertification and outcome studies may become important forces.^40,41 Improved physician performance with newer technologies and better resource utilization may decrease the cost of health care.^42,43

Styles of presentation

Numerous theories exist about the ideal method for adult learning and focus on the way information is conveyed. The success, both educationally and commercially, of CME is dependent on the extent to which adult learning is achieved in an aesthetically appealing manner.

Problem solving

The primary goal for a CME activity should be to meet the needs of the consumer. The ideal learning method for a professional medical practitioner is known to be problem solving.^44,45 This is especially true for "long-distance" learning in rural areas, where stress is high and the incentives to learn are relatively low.⁴⁶ Structured information, such as curriculum modules, is valuable in this setting, so that responses can be used to modify the information offered in subsequent presentations.

Practice medical education

Caplan⁴⁷ makes clear the notion that any form of mandatory education for physicians must be individualized to fit their needs or it will be met with opposition. If evaluation is planned, the most effective approach is structured self-assessment. Properly structured self-assessment has been validated enough to be acceptable to both governmental regulators and third party payers. Watts ⁴⁸ echoes this concept and describes the ideal mandatory CME as PME (practice medical education), which requires the incorporation of learning into the context of clinical activity. The superior quality of learning from practice is recognized in other professions. Judges and airline pilots recognize the unique structure of their tasks and the learning that evolves from each application. ⁴⁹ In medical practice, case study, peer review, and quality assurance data collections are examples of potential practice-based CME tools.

Auditing

Multimedia computers and simulators are tools that may move practice-based learning from the theoretical to the practical in the future. Al-Shehri et al.⁵⁰ and Cervero⁴⁹ further expand this approach to adult learning for the physician. They emphasize not only learning in a clinical situation but also learning from a clinical situation. Audits, retrospective series review, and continuous quality assurance activities are potential tools. The review of information that is now easily accessible from computer databases makes these learning tools all the more powerful. Auditing as a learning tool can be done randomly; however, more interest is generated when the audit has a focus. For instance, when the characteristics of a pathological process or health care delivery system are identified, clinical problem solving occurs. ⁵⁰ Audit methods that identify the needs of patients are particularly powerful learning tools.^8,9,51

Self-directed learning

Doyle⁵² identifies the needs of physicians in smaller states and rural areas of large states with the concept of self-directed learning. The physician is presumed to be an educated individual with more than 20 years of formal education. The natural curiosity of the educated mind is a potent stimulus for continued learning if supported by some structure. Such a learning program exists at the University of Arkansas Medical Center.

Some adult learning skills are accentuated by the format of presentation.^53,54 Self-learning requires either structure or support. Input into the program from outside a physician's specialty or even outside of the physician community is an enhancement to adult learning.⁵⁵ The first source of information about need is from the target physician pool and is followed by multidisciplinary team evaluation about how to deliver these needs. These ideas reflect the evolving concept that there are needs of physicians that are reflected in their practice and could be improved with ongoing education programs.

Role of computers, video, and multimedia

The concept of multidisciplinary input for CME easily incorporates the use of computers with multimedia capability. Jackson et al.⁵⁵ describes a number of programs sponsored by the organizations within the field of Emergency Medicine, which are made available "on-line" to members. Utilization and consumer satisfaction are high. Despite the lack of a structured format (physicians are allowed to review what they think they need from large amounts of available data), the passing level on a post-CME activity test was greater than 80%. Piemme ⁵⁶ reviewed and enumerated the immense amount of on-line information available to physicians, including the bibliographic databases, journals, and full-text activities, but notes that none are presently certified for formal CME credit. Piga et al.⁵⁷ describes the powerful impact of on-line video capabilities via personal computers. Barber⁵⁸ further expands the potential of video. He proposed a complex interactive learning style utilizing the review of videotape of teaching of clinical activity by peers with didactic feedback. The review of videotapes did not change the content of teaching, but it did change the style of delivery in subsequent presentations of the same material. Postcourse evaluations revealed an improvement of the overall teaching. This may have potential application in Anesthesiology with the expanding use of simulators, as is the case with airline pilots. Interactive videotape as a CME tool was prospectively studied for Family Practice by Premi and Shannon. ⁵⁹ A defined clinical situation was chosen, and a pretest was given. The physicians then were divided into two groups: one group was asked to study independently and the other group was given a specific group of journal articles to accompany a teaching video. Significantly better improvement occurred in the video-reference group. Bearon et al. ⁶⁰ expand on the advantages of defined video and hard-copy references for learning in a program for CME in the field of Gerontology created at Duke. Review of the video for content, accuracy, and effective delivery was performed by a multidisciplinary group. The combination of the video content with the multidisciplinary input was felt to enhance the learning in a synergistic manner. Briley et al. ⁶¹ report on the use of computer image transfer for learning at sites peripheral to a tertiary center for neuroradiology. Image transfer libraries for anesthesiology located on World Wide Web sites are just beginning to develop on the Internet, with similar educational potential.

Style of presentation

The format of CME courses varies. The speakers range from honorary invitees ("Rovenstine Lecture") to peer-selected individuals (Refresher Course) to those selected by course directors. Panel discussions often combine experts. Some CME presentations involve first-time presentation of scientific research (abstracts), and part of the CME is the interaction about the validity, merit, and future directions for the research. Scientific Exhibits are put together for large meetings to show new technology. Peer review for scientific merit is done during the selection process. This is not the case with the commercial exhibits of drug and technology products at major meetings, whose express purpose is scientific advertising. Use of a coached moderator has a favorable impact on audience satisfaction and outcome assessment by exit interview. ⁶²

Style of delivery also is influenced by choices of audiovisual support.⁶³ The education literature clearly establishes that adults learn more when the information is presented using multiple modalities. The combination of verbal communication with visual information (35-mm slides, written syllabus material) is clearly superior to either component individually. This remains the most popular style of presentation of physician CME. Resistance to alternatives is probably based on lack of prior experience with other styles. ⁶³ Computer-generated slides allow instantaneous generation of audiovisual support during a presentation. Multimedia audiovisual support will become more common as it becomes possible to instantaneously call up pre-prepared slides, video clips, and sound clips. The audience's role will become more interactive. The development of the computer-based Audience Response System will allow the audience to interact directly with the speaker's audiovisual materials. Creation of CME syllabus material will not disappear, but may evolve toward CD-ROM or home pages on the World Wide Web of the Internet.

Evaluation of CME

The mandatory nature and cost of CME activities necessitates evaluation of the CME product. A number of approaches have been described and criteria for quality have been defined.

Needs assessment

The ACCME mandates that for accreditation for AMA/PRA credit hours, CME activities must have a pre-presentation statement of objectives. Evaluation, through a systematic analysis of retrospective needs assessment given to each attendee, is based on their response regarding the success of the presentation and the potential for improvement. ⁶⁴ In this way, the evaluation of the CME entails methodically determining the extent to which the stated objectives were met.⁶⁵

Vanek⁶⁶ carried this needs assessment a step further by identifying the target group for several physician meetings prospectively, mailing a questionnaire to the potential participants, and using the information to modify presentations. Combined with the postcourse evaluation of outcome, the information was used to modify subsequent versions of the meeting. The cost was $1200.00 per meeting, which was 5% to 10% of the meeting's administrative budget. If this process were implemented as a standard routine and combined with brochure mailings, the cost would be reduced considerably.

Swanson and Jennett⁶⁷ describe a needs assessment process whereby refresher course goals, objectives, and physician-perceived learning needs were evaluated and subsequent presentation of this course was modified to include didactic material and case presentations. Those attending the first version of the course were asked specific questions about their most common treatments of a variety of clinical situations. Common errors were added as case material for the subsequent session. Sherman and Lambiase ⁶⁴ describe needs assessment sources beyond the individual participant, such as licensure, credentials, board certification, quality assurance, peer review, and evolving new technology. Sources for this information include the American Heart Association, medical schools, the AMA (through ACCME), the federal government (through Medicare), and the states (through licensure). This idea is a logical continuation of Continuing Education Systems Project (CESP) formed by the Association of American Medical Colleges, which suggested that regional medical centers provide and monitor CME. ⁶⁸ Although not fully accepted, this process involved more than 100 different criteria of a CME activity from notification, through content, to postactivity evaluation. The use of quality indicators and external review to improve ongoing CME activities is integral. ¹⁴ The state of Connecticut accredits hospitals to provide CME with its own set of criteria that also meet ACCME standards.²⁷

CRISIS criteria

A modification of the needs assessment with the acronym CRISIS was developed in 1982 by Harden and Laidlaw.⁶⁹ The categories are indicated by the letters: Convenience, Relevance, Individualization, Self-assessment, Interest, Speculation, and Systematic coverage. This allows a systematic approach to needs assessment and the evaluation of outcome comments by participants. Convenience was defined by location, timing, and speed of delivery. Relevance is determined after the fact by the participant.

Precourse surveys and use of postcourse evaluation for subsequent programs should improve this variable. Assessment of relevance by physicians is often most focused on priorities in clinical care of patients. The extent to which didactic teaching provides this need will determine relevance. Individualization is determined by the extent to which the course meets the needs of the practice style of the participant. Self-assessment is a tool for evaluation that requires participation of course registrants. It typically involves a question from the speaker, a response by the audience, and some discussion of the outcome. This has become electronic with the Audience Response System. A speaker asks a question, each participant in the audience has a keypad that feeds answers into a database on a small personal computer, and the answer is displayed electronically. The purpose of self-assessment is to get immediate feedback about the actual learning occurring during a given presentation and to use this information to further increase learning. Interest as an evaluation tool is obvious. The kind of response to subject material and brochure covers described earlier fits in this category. Interest will directly determine registration. Speculation is that part of learning where the teacher stimulates further ideas in related topics and interest in additional learning in the same area. Physicians are scientists, and the extent to which a given presentation can increase their intellectual curiosity is the extent to which success in this area is measured. Systematic refers to the instinctive need of participants to have the course presented in an organized fashion. Subjects should flow in a logical sequence; there should be minimal repetition and as few omissions in key areas as possible.

Levine et al.⁷⁰ fractionated needs assessment, focusing on participant satisfaction data, administrative data, and attitudinal data as they influence outcome. Evaluation of CME programs also can be accomplished by groups of CME providers. To improve outcome they disseminate information, measure the rate of adoption of the material, and evaluate the change in an outcome variable. ⁷¹ With increasing emphasis on outcome and economic performance, this method may become the standard for evaluation of CME.

In our era of consumerism, it should not be surprising to see complex advertising techniques involved in publicity and evaluation of CME. One approach uses the classic focus groups of new-product development for the needs assessment and evaluation of CME activity. ⁷² The key role of local CME supervisors in directing the curriculum of CME toward the needs of the community is recognized.⁷³ Because not all CME is external, there needs to be evaluation of self-directed CME. The Royal College of Physicians and Surgeons of Canada⁷⁴ has developed such a program, called The Maintenance of Competence Program. The primary purpose of the program is to provide education, and the assessment is a self-completed diary. The participant provides feedback to the program about what was learned from the presentation and also whether the content will modify his or her practice.

The future of CME

Multimedia and the internet

Some future directions of CME have already been discussed, and some elements, such as the use of computers in CME, already exist. Access to medical databases has become routine.⁷⁵ Chao⁷⁶ reviewed available CME software and makes a case for steady expansion. As medical practice steadily becomes more technological and the use of computers in clinical work becomes more common, computer-generated CME logically follows. Ideally, software should address clinical problems, have a reasonable cost, and be easily updated. Comprehensive materials may utilize CD-ROM and have the advantage of huge capacity, but they also have the disadvantage of not being able to update the information without producing a new CD-ROM disk. Vanek et al. ⁷⁷ establish the value of computer networks for precourse needs assessments, which decreases cost and provides improved response rates. Staab⁷⁸ presents the role of multimedia presentation to improve adult learning. Interactive video with computer enhancement via the Internet is an example.⁷⁹ Pre- and post-testing can validate the educational benefits of material available on-line. Interactive CME via computer videodisk also is possible, although interactivity is much less.⁸⁰ The use of multimedia for learning through networks is even more beneficial, because the learning is interactive among the students.⁸¹ Folberg et al.⁸² present the evolving technology of compact disk and videodisk, and their interactive versions, as the teaching tools of the future for highly technical learning, such as eye surgery. In this setting, choices elicit responses and can be modified to improve technique outside the operating room. Virtual hospitals exist on the Internet and via modem at numerous sites. ^2,21,83,84 Virtual schools also are possible,⁸³ as are the presentation and evaluation of new products and technology on-line.³ Because of the multimedia capability of the World Wide Web, different data formats such as video, audio, physical diagnosis, algorithms, teaching files, X-ray films, and even whole textbooks can be available immediately, either alone or in combinations.

Anesthesia simulators

The present and future of CME in Anesthesiology will be influenced heavily by the development and increasing availability of full-scale anesthesia simulators. Simulation of clinical care in anesthesiology parallels simulation training in the airline industry, where a combination of mechanical, audiovisual, and intellectual resources must be used simultaneously for problem solving. ^85,86 This type of training not only facilitates adult learning from multimedia presentation, but also may actually improve crisis-management skills.⁸⁶ In groups of fully trained anesthesiologists, prior simulation of critical incidents improved the technical response to subsequent simulated crisis scenarios compared to individuals without prior exposure.^87,88 Other fully trained anesthesiologists who were asked to compare simulator scenarios to critical events that occurred in the operating room found the simulations to be highly realistic in a majority of subjects.^89,90 In addition to technical experience, simulator training is ideally suited to the evaluation of learning and performance.⁹¹ Ideal characteristics of CME via simulator take advantage of learning in the context of clinical care, which is easily and safely repeated, driven by commonly known data, and technologically feasible.⁹² Virtual reality, artificial intelligence, and clinical simulators are those areas where future technology will have a heavy impact on adult education and CME.

State	Title	Location	Comments
New York	Physician-Prescribed Education Program	Syracuse University	At Site and Home Practice (38)
Oregon	Individualized Physician Renewal Program	Home Practice Site
Maryland	Physician Remedial Focused Educational Program	University of Maryland	Targeted CME
Colorado	Personalized Education Program for Physicians	Home Practice Site	Targeted CME. Accepts Self-Referral
North Carolina	Clinical Enhancement Program	East Carolina School of Medicine	Consultation and Rural Support
California	Professional Competence Assurance Program	Administrative	Record Review-Drive Directed CME
Iowa (39)	State Medical Board	Administrative	Directed CME and On-Site Follow-Up

References

1 R.D. Fox, Theoretical foundations of continuing education in the health professions, J Continuing Ed Health Profess, Vol. 10, 1990, 71-72

2 S. Misra, W.O. Robertson, CME needs assessment, a statewide survey, J Continuing Ed Health Profess, Vol. 14, 1994, 119-125

3 J.K. Stross, T.J. DeKornfeld, A formal audit of continuing medical education activity for license renewal, JAMA, Vol. 264, 1990, 2421-2423

4 M.S. Eslami, the CSMS role in continuing medical education, Calif Med, Vol. 55, 1991, 179

5 J.E. Verby, A new model for CME, Acad Med, Vol. 67, 1992, 316-317

6 L. Curry, J.F. Wergin, Educating professionals, responding to new expectations for competence and accountability, J Continuing Ed Health Profess, Vol. 13, 1993, 251-255

7 A.D. Hewson, Continuing medical education in obstetrics and gynaecology, the challenge of the nineties, Aust N Z J Obstet Gynaecol, Vol. 31, 1991, 249-253

8 D.K. Wentz, Continuing medical education at a crossroads, JAMA, Vol. 264, 1990, 2425-2426

9 D.K. Wentz, A.M. Osteen, M.I. Gannon, Continuing medical education, unabated debate, JAMA, Vol. 268, 1992, 1118-1120

10 J.T. Parboosingh, S.T. Gondocz, The maintenance of competence program of the Royal College of Physicians and Surgeons of Canada, JAMA, Vol. 270, 1993, 1093

11 G.A. Reuther, AOA continuing medical education, J Am Osteopath Assoc, Vol. 90, 1990, 1020-1026

12 R.M. Nambiar, Recertification, Ann Acad Med, Vol. 21, 1992, 721-722

13 J. Trautlein, J.T. Ziegenfuss, A. Pelberg, W. Wenner, T. Casale, The fellowship program in quality assurance and utilization review, Am Coll Utiliz Rev Physicians, Vol. 4, 1990, 107-113

14 F.A. Anderson Jr, B. Wheeler, R.J. Goldberg, D.W. Hosmer, A. Forcier, N.A. Patwardhan, Changing clinical practice, Arch Intern Med, Vol. 154, 1994, 669-677

15 M.B. Kapp, Survey of continuing medical education programs in legal liability and risk management, J Continuing Med Ed Health Profess, Vol. 14, 1994, 114-122

16 P. Riis, J Continuing Ed Health Profess, Vol. 10, 1990, 111-115

17 J.C. Leist, R.E. Kristofco, The changing paradigm for continuing medical education, impact of information on the teachable moment, Bull Med Libr Assoc, Vol. 78, 1990, 173-179

18 F. Rosner, J.A. Balint, R.M. Stein, Remedial medical education, Arch Intern Med, Vol. 154, 1994, 274-282

19 R. Nelson, Recreditation through personalized CME, Iowa Med, Vol. 83, 1993, 381

20 D.A. Davis, P.E. Mazmaniam, On CME and the reform of medical education, Acad Med, Vol. 69, 1994, 207-208

21 R.S. Henderson, Continuing Education Committee of Anesthetists of New Zealand (CECANZ), the first five years, Anaesth Intensive Care, Vol. 20, 1992, 211-214

22 W.O. Robertson, S. Misra, Correspondence, assessing CME needs assessment, J Continuing Med Ed Health Profess, Vol. 14, 1994, 535-536

23 C.E. Koop, J Continuing Med Ed Health Profess, Vol. 10, 1990, 103-109

24 W. Levinson, D. Roter, The effect of two continuing medical education programs on communication skills of practicing primary care physicians, J Gen Intern Med, Vol. 8, 1993, 318-324

25 R.V. Harrison, The influence of brochure covers on CME enrollments, J Continuing Ed Health Profess, Vol. 11, 1991, 265-269

26 D.N. Kerr, S.A. Jones, C.S. Easmon, Continuing medical education, experience and opinions of consultants, BMJ, Vol. 306, 1993, 1398-1402

27 P.J. Toghill, Beyond postgraduate training, Postgrad Med J, Vol. 70, 1994, 63-64

28 P.J. Toghill, Continuing medical education for physicians, J R Coll Physicians, Vol. 28, 1994, 155-156

29 R.V. Harrison, J.K. Stross, Acad Med, Vol. 68, 1993, 841-842

30 T. Randall, New guidelines expected in 1991 for relationship on continuing education, financial support, JAMA, Vol. 264, 1990, 1080

31 K.P. Bunnell, New standards for commercial support of CME, J Cancer Ed, Vol. 7, 1992, 203-207

32 B. Paccagnella, Alternatives and options for funding continuing medical education, a debate, Med Ed, Vol. 24, 1990, 74-77

33 E. Odenbach, Collaboration with commercial interests in continuing medical education, J Continuing Ed Health Profess, Vol. 10, 1990, 293-301

34 B. Canavan, Forging the alliance, J Continuing Ed Health Profess, Vol. 10, 1990, 123-128

35 H.A. Holm, B.O. Hoftvedt, A. Lie, E. Skoglund, Continuing medical education, the Norwegian way, J Continuing Ed Health Profess, Vol. 13, 1993, 77-83

36 F. Difford, R.C. Hughes, Br J Gen Pract, Vol. 42, 1992, 290-293

37 I. Forgaes, Continuing education and medical schools, Med Ed, Vol. 26, 1992, 85-86

38 M.R. Schwarz, Acad Med, Vol. 69, 1994, 470-471

39 M.S. Watts, Continuing medical education and public policy in an era of health care reform, J Continuing Ed Health Profess, Vol. 13, 1993, 247-248

40 P.L. Walsh, Planning and developing programs systematically, J Med Ed, Vol. 56, 1981, 132-151

41 K.A. Hecht, CME in the decade of health care reform, J Continuing Ed Health Profess, Vol. 13, 1993, 211-214

42 S.J. Jay, J.G. Anderson, Continuing medical education and pubic policy in an era of health care reform, J Continuing Ed Health Profess, Vol. 13, 1993, 195-209

43 D.E. Moore, Evolving approaches to continuing medical education, efforts to enhance the impact on patient care, J Med Ed, Vol. 56, 1981, 87-114

44 J.H. Shin, R.B. Haynes, M.E. Johnston, Effect of problem-based, self-directed undergraduate education on life-long learning, Can Med Assoc J, Vol. 148, 1993, 969-976

45 S. Crandall, How expert clinical educators teach what they know, J Continuing Ed Health Profess, Vol. 13, 1993, 85-98

46 C.E. Engel, E. Browne, P. Nyarango, Problem-based learning in distance education, a first exploration in continuing medical education, Med Ed, Vol. 26, 1992, 389-401

47 R. Caplan, The evolution of individualized CME, Iowa Med, Vol. 81, 1991, 48-49

48 M.S. Watts, J Continuing Ed Health Profess, Vol. 10, 1990, 129-136

49 R.M. Cervero, The importance of practical knowledge and implications for continuing education, J Continuing Ed Health Profess, Vol. 10, 1990, 85-94

50 A. Al-Shehri, I. Stanley, P. Thomas, Continuing education for general practice 2, systematic learning from experience, Br J Gen Pract, Vol. 43, 1993, 249-253

51 D.K. Wentz, M.I. Gannon, A.M. Osteen, Continuing medical education, JAMA, Vol. 264, 1990, 836-840

52 L.L. Doyle, Continuing medical education the quiet revolution, J Arkansas Med Soc, Vol. 90, 1994, 533-534

53 S.J. Grosswald, Designing effective education activities, J Med Ed, Vol. 56, 1981, 174-196

54 S.J. Grosswald, E. Suter, G.R. Seppala, D.B. Walthall, Improving continuing education in diverse health care settings, J Med Ed, Vol. 56, 1981, 338-360

55 B.J. Jackson, B.J. Hays, C.C. Robinson, Multiple delivery methods for an interdisciplinary audience, assessing effectiveness, J Continuing Ed Health Profess, Vol. 10, 1990, 59-69

56 T.E. Piemme, Computers and medical education, J Continuing Ed Health Profess, Vol. 12, 1992, 89

57 A. Piga, R. Bascioni, M. DiGiuseppe, R. Cellerino, Continuing medical education by the videotex system, Rays, Vol. 16, 1991, 513-517

58 S.G. Barber, Postgraduate teaching audit by peer review of videotape recordings, Med Teacher, Vol. 14, 1992, 149-157

59 J. Premi, S.I. Shannon, Education in the information age, randomized controlled trial of a combined video-workbook educational program for CME, Acad Med, Vol. 68, 1993, 513-515

60 L.B. Bearon, D.H. Hickman, A.H. Amara, Video journal club, combining video evaluation and continuing education, Gerontologist, Vol. 33, 1993, 415-418

61 M.S. Briley, M. Hoare, D. Dobson, P. Lanslow, Imlink and continuing medical education, the use of an image transfer system to broadcast teaching cases nationally, Br J Radiol, Vol. 67, 1994, 453-455

62 T.L. Yarboro, Guidelines for the moderator, J Natl Med Assoc, Vol. 82, 1992, 49-51

63 P.A. Jennett, R.W. Swanson, Traditional and new approaches to CME, perceptions of a variety of CME activities, J Continuing Ed Health Profess, Vol. 14, 1994, 75-82

64 C.D. Sherman, P. Lambiase, Continuing medical education in the United States, a critique, Eur J Cancer, Vol. 29A, 1993, 784-787

65 T.J. Sork, Theoretical foundations of education program planning, J Continuing Med Ed Health Profess, Vol. 10, 1990, 73-83

66 E.P. Vanek, W.D. Carey, M. Sesic, D.M. Jackman, B. Fleshler, Value of needs assessment surveys, J Continuing Ed Health Profess, Vol. 14, 1994, 224-231

67 R.W. Swanson, P.A. Jennett, Observations on the content of a formal CME course before and after needs assessment, J Continuing Ed Health Profess, Vol. 12, 1992, 235-239

68 E. Suter, J.S. Green, S.J. Grosswald, K.A. Lawrence, D.B. Walthall III, C., Zeleznik, Introduction, defining quality for continuing education, J Med Ed, Vol. 56, 1981, 1-31

69 R.M. Harden, J.M. Laidlaw, Effective continuing education, the crisis criteria, Med Ed, Vol. 26, 1992, 408-422

70 H.G. Levine, D.E. Moore, F.C. Pennington, Evaluating continuing medical education activities and outcome, J Med Ed, Vol. 56, 1981, 197-217

71 P.E. Mazmanian, R.B. Williams, C.E. Desch, R.E. Johnson, Theory and research for the development of continuing education in the health professions, J Continuing Med Ed Health Profess, Vol. 10, 1990, 349-365

72 J. Tipping, J. Tennenbaum, The use of focus groups as a tool for CME program evaluation, J Continuing Ed Health Profess, Vol. 13, 1993, 117-122

73 W.C. Felch, CME and the art of medicine, J Continuing Ed Health Profess, Vol. 11, 1991, 169-173

74 J.T. Parboosingh, S.T. Gondocz, The maintenance of competence (MOCOMP) program, motivating specialists to appraise the quality of their continuing medical education activities, JCC, Vol. 36, 1993, 29-32

75 P.R. Manning, Continuing education needs of health care professionals, Bull Med Libr Assoc, Vol. 78, 1990, 161-164

76 J. Chao, Continuing medical education software, a comparative review, J Fam Pract, Vol. 34, 1992, 598-604

77 E.P. Vanek, E. Pomiecko, Using a computerized information network for assessing programming needs, J Continuing Ed Health Profess, Vol. 14, 1994, 115-118

78 E.V. Staab, Electronic media for education and communication in radiology, the saga continues, Radiographics, Vol. 13, 1993, 1181-1182

79 C.L. Hampton, P.E. Mazmanian, T.J. Smith, The interactive videoconference, an effective CME delivery system, J Continuing Ed Health Profess, Vol. 14, 1994, 83-89

80 R.V. Jones, Continuing professional education, Postgrad Med J, Vol. 69, 1993, S91-S93

81 C.J. Scott, Applied adult learning theory, broadening traditional CME programs with self-guided, computer-assisted learning, J Continuing Med Ed Health Profess, Vol. 14, 1994, 91-99

82 R. Folberg, L.K. Dickinson, R.A. Christiansen, J.S. Huntley, D.G. Lind, Interactive videodisc and compact disc-interactive for ophthalmic basic science and continuing medical education, Ophthalmology, Vol. 100, 1993, 842-850

83 P.A. McKee, R. Whang, A college of medicine without walls, continuing medical education, J Oklahoma State Med Assoc, Vol. 85, 1992, 15-16

84 J.R. Galvin, W.E. Erkonen, D.L. Lacey, D.M. Santer, The virtual hospital, a link between academia and practitioners, J Continuing Ed Health Profess, Vol. 14, 1994, 130

85 M.M. Kurrek, K.J. Fish, Anaesthesia crisis resource management training, an intimidating concept, a rewarding experience, Can J Anaesth, Vol. 43, Iss. 5 Pt 1, 1996, 430-434

86 S.K. Howard, D.M. Gaba, K.J. Fish, G. Yang, F.H. Sarnquist, Anesthesia crisis resource management training, teaching anesthesiologists to handle critical incidents, Aviat Space Environ Med, Vol. 63, 1992, 763-770

87 A. De Anda, D.M. Gaba, Role of experience in the response to simulated critical incidents, Anesth Analg, Vol. 72, 1991, 308-315

88 V. Chopra, B.J. Gesink, J. de Jong, J.G. Bovill, J. Spierdijk, R. Brand, Br J Anaesth, Vol. 73, 1994, 293-297

89 D.M. Gaba, A. DeAnda, A comprehensive anesthesia simulation environment, re-creating the operating room for research and training, Anesthesiology, Vol. 69, 1988, 387-394

90 V. Chopra, F.H. Engbers, M.J. Geerts, W.R. Filet, J.G. Bovill, J. Spierdijk, The Leiden anaesthesia simulator, Br J Anaesth, Vol. 73, 1994, 287-292

91 D.M. Gaba, Anesthesiology, Vol. 76, 1992, 491-494

92 J.M. Davies, R.L. Helmreich, Simulation, Can J Anaesth, Vol. 43, 1996, 425-429