Lessons We Don't Learn: A Study of the Lessons of Disasters, Why We Repeat Them, and How We Can Learn Them

Abstract

Emergency responders intervene before and during disasters to save lives and property. The uncertainty and infrequency of disasters make it hard for responders to validate that their response strategies will be effective, however. As a result, emergency response organizations use processes for identifying and disseminating lessons in hopes that they and others will be able to learn from past experience and improve future responses. But the term "lessons learned" may be a misnomer. Anecdotal evidence suggests mistakes are repeated incident after incident. It appears that while identifying lessons is relatively straightforward, true learning is much harder - lessons tend to be isolated and perishable, rather than generalized and institutionalized. That we see problems persist is a serious concern; as emergency response missions expand to include broader homeland security responsibilities, the ability to capitalize on experience is ever more important. This article reports the results of a qualitative study of both the lessons themselves and the efficacy of the processes by which responders hope to learn them.

On February 23, 2006, in a press conference to release the White House report on lessons learned from Hurricane Katrina, Assistant to the President for Homeland Security and Counterterrorism Frances Townsend said "[The president] demanded that we find out the lessons, that we learn them and that we fix the problems, that we take every action to make sure America is safer, stronger and better prepared." The lessons Townsend called out in her briefing concerned planning, resource management, evacuation, situational awareness, communications, and coordination. No one in the emergency response community was surprised. We know these are the problem areas. We knew they would be before Katrina ever hit the Gulf coast. Why? Because we identify the same lessons again and again, incident after incident.

In fact, responders can readily predict the problems that will arise in a major incident and too often their predictions are borne out in practice. Even a casual observer can spot problems that recur: communications systems fail, command and control structures are fractured, resources are slow to be deployed. A quick perusal of the reports published after the major incidents of the past decade quickly shows this to be true. Consider the following:

Hurricane Katrina, 2005 In terms of the management of the Federal response, our architecture of command and control mechanisms as well as our existing structure of plans did not serve us well. Command centers in the Department of Homeland Security (DHS) and elsewhere in the Federal government had unclear, and often overlapping, roles and responsibilities that were exposed as flawed during this disaster...This lack of coordination at the Federal headquarters-level reflected confusing organizational structures in the field...Furthermore, the JFO [Joint Field Office] staff and other deployed Federal personnel often lacked a working knowledge of NIMS [the National Incident Management System] or even a basic understanding of ICS [Incident Command System] principles.

- From The Federal Response to Hurricane Katrina Lessons Learned, 2006: 52

September 11 attack, 2001 It is a fair inference, given the differing situations in New York City and Northern Virginia, that the problems in command, control, and communications that occurred at both sites will likely recur in any emergency of similar scale. The task looking forward is to enable first responders to respond in a coordinated manner with the greatest possible awareness of the situation...Emergency response agencies nationwide should adopt the Incident Command System (ICS).When multiple agencies or multiple jurisdictions are involved, they should adopt a unified command. Both are proven frameworks for emergency response.

- From The 9/11 Commission Report, 2004: 315, 397

Oklahoma City bombing, 1995 The Integrated Emergency Management System (IEMS) and Incident Command System (ICS) were weakened early in the event due to the immediate response of numerous local, state and federal agencies, three separate locations of the Incident Command Post (ICP), within the first few hours, and the deployment of many Mobile Command Posts (MCPs), representing support agencies.

- From the After Action Report: Alfred P. Murrah Federal Building Bombing, 2003: 3

Hurricane Andrew, 1992 The Committee heard substantial testimony that the post-disaster response and recovery to Hurricane Andrew suffered from several problems, including: inadequate communication between levels of government concerning specific needs; lack of full awareness of supply inventories and agency capabilities; failure to have a single person in charge with a clear chain of command; and inability to cut through bureaucratic red tape.

- From the Governor's Disaster Planning and Response Review Committee Final Report, 1993: 60

As these statements reveal, we repeatedly confront command and control issues in large incidents. These are but a few examples from dozens of reports that cite the need for sound command structures. Somehow, though, we fail to learn this and other crucial lessons that have been identified in after-action reports for decades. The central concerns of this paper are why that is so and how we can improve. We report here on an exploratory investigation that targets six research questions.

1. Is it true that lessons recur?

2. What lessons are persistently identified?

3. Why do these lessons continue to be identified as important?

4. Why are these lessons so hard to learn? (That is, why do agencies have difficulty devising and implementing corrective actions once lessons are identified?)

5. How do lessons-learned processes work?

6. How can they be improved?

We believe that by explicitly identifying persistent challenges, responders may be better attuned to these challenges and more able to address them in their planning and training processes. Likewise, by better understanding why these challenges remain unresolved, responders may be able to adapt their lessons-learned processes to better support behavioral change and improvement. To these ends, we have conducted a qualitative analysis of response organizations' perspectives on lessons and learning. The next section describes the context of emergency response learning. We then explain our investigative approach. Following that we present and discuss our findings about what lessons responders struggle with most and what learning approaches they use. We conclude with recommendations for improving these processes.

Disasters and Learning

Disasters are devastating natural, accidental, or willful events that suddenly result in severe negative economic and social consequences for the populations they affect, often including physical injury, loss of life, property damage and loss, physical and emotional hardship, destruction of physical infrastructure, and failure of administrative and operational systems. Emergency managers and responders are responsible for intervening before and during such events, to minimize the harm disasters cause and to restore order. The large scale, high complexity, profound urgency, and intense scrutiny that attend disasters provide a powerful motivation for responders to be good at response.

To address this challenge, responders use their experience to develop systematized strategies they can follow when the chaos of disaster erupts. At the same time, the infrequency with which disasters occur makes it hard for responders to test and improve their strategies, to ensure that they can be counted on to mitigate threats and hazards predictably and to resolve their consequences effectively. The appeal of learning from experience - both to avoid duplicating mistakes and to be able to repeat successes - is widely perceived, and many organizations across the emergency response disciplines have formal procedures for identifying, documenting, and disseminating lessons from incidents in hopes that they and others will be able to learn from past experience and improve future responses.

Various mechanisms for sharing experience have emerged. These mechanisms are generally termed "lessons-learned" processes, and include tools like in-progress reviews, after-action reviewing and reporting, "hotwashes," and various kinds of debriefings. While these processes vary, they have the common goal of sharing performance information in order to prevent the recurrence of adverse events and actions and to better contend with situations and problems that are likely to arise again. Most processes involve some version of three core components: 1. Evaluating an incident (through systematic analysis of what happened and why); 2. Identifying lessons (strengths to be sustained and weaknesses to be corrected); and 3. Learning (specifying and inculcating behavioral changes consistent with the lessons).

Examples of lessons-learned systems abound. One of the best known is the U.S. Army's After Action Review (AAR), a comprehensive reflective learning process developed in the 1970s. ¹ Many emergency responders follow the AAR template to a greater or lesser extent, formally or informally. Post-incident reporting is a common practice whereby an agency or set of agencies documents what happened during a disaster or exercise. These reports usually include accounts of actions and results, as well as potential remedies to problems encountered. While these reports are often used internally by the agencies that generate them, they are often written in isolation by a single agency, rather than through a coherent inter-agency process. There are various collections of lessons that have been compiled for broader distribution. Prominent examples include the Wildland Fire Lessons Learned Center collection and the recently established Lessons Learned Information System (developed by the memorial Institute for the Prevention of Terrorism and sponsored by the Department of Homeland Security).

Despite these widespread activities, however, the term "lessons learned" is often a misnomer. Our experience suggests that purported lessons learned are not really learned; many problems and mistakes are repeated in subsequent events. It appears that while review of incidents and the identification of lessons are more readily accomplished, true learning is much more difficult. Reports and lessons are often ignored, ² and even when they are not, lessons are too often isolated and perishable, rather than generalized and institutionalized.

Methodology

To determine whether or not our instinct is correct - that emergency response organizations find it difficult to learn certain lessons - and to better understand why this is the case, we decided to conduct an exploratory analysis. We used three qualitative approaches in our investigation: interviews, a review of documents, and a focus group retreat. We began our study with a series of informal interviews with experienced emergency responders to confirm the face validity of our hypothesis that important lessons are repeatedly identified and to verify that this was a compelling concern for emergency responders. We then reviewed reports produced following incidents to discover and classify lessons that are identified repeatedly. We included reports from large incidents of all types that occurred within the past two decades. We excluded reports from military operations and from exercises. In many cases, individual organizations prepare their own reports, and so there are often several reports available for a given incident. In these cases, we focused on the reports prepared at the government level, rather than at the agency level, often by the department (or office) of emergency management, but sometimes by an independent analyst or commission. The reports we reviewed are listed in Appendix A.

Some reports were very general, identifying major issues and general lessons. Many, though, were very detailed and the descriptions and explanations provided were very particular to the incident at hand. Because we are interested in high-level, cross-cutting lessons, we confined our examination to lessons that were called out in an executive summary (if provided) or that were in some way highlighted as significant in the body of the report. Our review of these reports can best be characterized as systematic, but informal. That is, we did not apply formal coding schemes or use sophisticated textual analysis methods. While this would certainly be an interesting avenue for further analysis, our purpose was to determine whether significant lessons were common across reports.

Finally, we convened a focus group of eleven expert incident managers who could reflect on the persistent concerns that arise during major disasters. Most participants were chief-level officers. All were from major U.S. municipalities. All participants had significant senior-level management experience dealing with large scale incidents. Examples of incidents they had managed include: Hurricane Katrina (2005), the Columbia space shuttle crash (2003), the anthrax and ricin attacks in Washington, D.C. (2001), the September 11^th attacks at both the World Trade Center and the Pentagon (2001), the crash of American Airlines Flight 587 (2001), the bombing of the Murrah Building in Oklahoma City (1995), the Northridge earthquake (1994), the World Trade Organization protests (1991), the Air Florida plane crash, as well as numerous other "civil" events such as presidential inaugurations, national political conventions, protests, major sports championships, Mardis Gras celebrations, and a multitude of natural disasters including wildfires, hurricanes, and tornados. Participants represented a range of emergency response disciplines including municipal and wildland firefighting, law enforcement, emergency medical services, urban search and rescue, and hazardous materials response. A list of the participants is provided in Appendix B.

During an intensive full-day retreat, we conducted a facilitated discussion to elicit the perspectives of these managers on our research questions. We had two primary objectives. First, we sought independent confirmation of the classes of lessons we discovered in our review of AAR's. To accomplish this we simply asked participants what major lessons seemed to come up repeatedly in their experience. Second, we wanted to elicit their beliefs about why these lessons were repeated rather than learned. We asked them a series of open-ended questions about why lessons are hard to learn, how lessons are identified and reported, and what mechanisms are used to prompt learning. Three note-takers independently documented the discussion that ensued. Participants were also afforded the opportunity to provide additional commentary to clarify or expand points they wanted to make.

Findings: What Lessons are Identified Repeatedly?

To reiterate, we sought to be systematic in our analysis, but this remains an exploratory investigation - a first step in an area we hope to probe further in a more targeted way. The findings we report in this section were garnered both deductively (proceeding from loosely-specified hypotheses) and inductively (in that new and unexpected insights surfaced and added to our inquiry). Our findings are admittedly subject to the biases inherent in subjective, qualitative research. We hope to mitigate this threat by citing the perspectives of our participants directly, so that the reader can "hear" how these individuals characterized the issues at hand. Thus we report here our synthesized findings accompanied by illustrations from the discussions we held.

Our review of AAR's bears out our hypothesis that lessons are repeatedly identified. Despite the disparity of the reports we reviewed, we found a striking consistency in major categories of lessons identified. Table 1 shows important topics that were addressed in several prominent incidents. While it is certainly the case that each incident had its own unique challenges, it was common to see problems characterized in similar ways across several incidents. It is also true that the response to some incidents appeared to go well while the response to others went badly, so that certain lessons were stated as successes to be repeated in some cases but as problems to be corrected in others. A detailed list of the lessons identified in a selection of reports for significant recent incidents is available from the authors.

Table 1. Common categories of lessons.

Correlation between After Action Reports from selected major incidents and significant issues addressed.

[Figure omitted. See PDF.]

We gain added confidence in our hypothesis that these lessons recur from the responses of our focus group. The focus group participants were easily able to identify lessons that emerge regularly from incident responses. There was a very high level of consensus among participants about what these lessons are, and the lessons they identified are very consistent with those we identified from our AAR review. The lessons our incident managers singled out as important and recurring pertain to five main areas: command, communications, planning, resource management, and public relations.

Uncoordinated Leadership

We asked our incident commander focus group "what problems do you see on every incident?" Several incident commanders immediately replied: unclear, multiple, conflicting, uncooperative, and isolated command structures. Every head in the room nodded agreement. Large incidents demand that robust command and control structures emerge out of the initial chaos that inevitably ensues when disasters strike. Large incidents also involve a multitude of agencies, each of which must direct its own resources. As a result, agency- and/or function-specific command structures proliferate. Since each agency has legitimate missions, responsibilities, and jurisdiction, each uses its command and control process to take charge, in a legitimate attempt to solve the problems the agency is supposed to solve. Absent an overarching command structure to which all participants subscribe, however, the result is duplicative and conflicting efforts. As one responder put it, "People ask 'who's in charge?' The response is usually, 'Of what?'" In fact, a coherent joint command structure often fails to emerge; our focus group specifically cited weak implementation of the incident command system (ICS) and poor understanding of unified command. A fire chief with extensive experience at the Katrina response gave a telling example: "In New Orleans, you couldn't go two blocks without running into somebody's incident command post. But there was no coordination between them. Everyone assumes there'll be a graduation up to some larger structure, but nobody knows how to get to that." At the same time, by using the term "command and control," we do not mean to suggest that structures are unitary, rigid, or static. In fact, successful management requires collaboration, flexibility, and adaptability across multiple diverse actors. This cannot be achieved anarchically, however; it requires that managers employ common philosophies and conventions.

What accounts for command problems, for failure to collaborate? Our emergency response experts cited three main culprits. First, they said, agencies lack the commitment to coordinate with each other. At best, they are unaware of what other agencies are doing and do not try to find out. At worst, they are unwilling to cooperate. This stems from a lack of trust between agencies and a lack of understanding across disciplines. Moreover, agencies often find themselves in competition. Day-to-day they fight with each other for scarce budget resources. This battle worsens during a major disaster when resources become even scarcer. Second, responders told us that the primary mechanism for resolving resource-allocation struggles, the Emergency Operations Center (EOC), is often ineffective. The delegates sent to EOCs are usually liaisons who lack decision-making authority, aren't respected, and/or don't get along with each other. They do not focus on how to make decisions together. Worse, large incidents spawn multiple EOCs that tend to be political and parochial - they will not exchange representatives to facilitate coordination. As a result, "turf battles" rage and distract incident managers from the real job at hand: mitigating the incident.

Finally, our experts told us, ICS is in common use, but it is not understood and implemented in a consistent manner. Generally, every discipline does their own form of ICS training and agencies train in isolation. Often this training is too simplistic to delve into the subtle skills of disciplined, team-based, decision making. Further, responders cannot be expected to learn the functions of incident management in the heat of an event. As one captain told us, "You can't grab 'regular' police officers and firefighters and take them away from handling the stuff they're handling to do incident management stuff. If they haven't already been training in logistics, it will take them a long time to figure it out, and they have other things to be worrying about." Yet, absent sound training, this is exactly what happens, with the needless result that recognized and well-developed incident management functions are carried out poorly. The reports cited in the introduction to this article bear out our focus group participants' claim that, as one manager put it, "Everyone agrees we need ICS, but we don't share one system."

Failed Communications

Our systems of command, control, and coordination are predicated on being able to communicate. As one expert told us, "For thirty years, we've said that communications is our biggest problem because it's a house of cards: When communications fails, the rest of the response fails." A major challenge of large disasters is that they destroy our physical infrastructure, including our communications equipment. The most recent example of this comes from Hurricane Katrina, which "destroyed an unprecedented portion of the core communications infrastructure throughout the Gulf Coast region... The complete devastation of the communications infrastructure left emergency responders and citizens without a reliable network across which they could coordinate." ³

But communications isn't entirely (or even fundamentally) a technology problem. We know how to build robust equipment and systems; as one participant noted, "CNN never goes down." And even sophisticated interoperable capability exists. But our response professionals pointed to an unwillingness to agree to a shared system, a lack of commitment to operate using this system, and a lack of discipline to use it correctly. As one chief pointed out, "We dump millions into hardware, but don't think about systems. Hardware will do anything you want. You've got to get people to agree on how to function with it." In short, technology is only an enabler; communicating requires that people are willing to share information with each other. This is not to say there are no important technological weaknesses in our communications systems. In part, communications deficiencies stem from gaps in research and development, from resource constraints, and from problems making some technologies broadly available. As one responder lamented, "We can talk to a rover on Mars, but we can't talk to someone inside a building." Despite being aware of the limitations and fragility of the infrastructure, we continue to lack contingency plans for how to communicate when technology fails (or is destroyed).

Weak Planning

Gaps in emergency plans cause serious problems when disaster strikes. Witness the evacuation problems experienced in New Orleans: Thousands of people had no way to leave the city on their own and no place to go, leaving them stranded in the face of Katrina. This problem was anticipated, yet the city's evacuation plan was woefully inadequate. While it mentioned evacuation, it lacked details about how evacuation would be conducted and who was responsible for the process, while some people who were assigned roles by the plan were unaware of their responsibilities. This is a prominent example, but not atypical of the response plans on which this nation relies. Plans are often simplistic and superficial, failing to provide enough detail to be actionable. Often plans cover the first hours or days of an incident, but do not consider long-duration responses or long-term recovery.

These gaps are a result of weaknesses in the planning process. The most fundamental problem to plague planning processes is a lack of commitment to plans across agencies and jurisdictions. While agencies may be at the table during the planning process, they may not buy in to the requirements needed to fully enact these plans. Alternatively, plans may be watered down to permit compromise, rather than requiring hard choices. Decisions about how work will get done are necessarily decisions about who has authority and who gets resources. These can be hard conflicts to resolve, and agencies often shirk making these hard choices when they are not perceived as immediately pressing. Worse, key agencies may be excluded from the planning process, even though the plan governs them or counts on their support. These problems are exacerbated by the fact that planning processes are typically infrequent, so plans become dated and do not incorporate lessons from recent events.

Ultimately these weaknesses go unnoticed because actual plans are not trained fully or exercised realistically. Plans are often developed by mid-level managers. Senior managers and political officials may have the plan on their shelves, but get no formal training on what is in it or how to use it. Similarly, plans are not disseminated to supervisors or training academies. When the time comes for implementation, those on the front lines don't know what the plan calls for.

Resource Constraints

Large-scale, long-duration incidents demand more resources - personnel, equipment, supplies, commodities, specialized capabilities - than any agency or government can keep on hand, so these resources must be obtained rapidly when a disaster occurs. This makes resource acquisition and management a major function of incident management. Unfortunately, while some materials are cached and pre-deployed, they are often inadequate to meet actual need. This means that resources must be obtained "real-time," but normal resource acquisition systems are too slow and are not designed to obtain large amounts of supplies rapidly. The capacity and flexibility of emergency requisition and purchasing procedures are uneven. Bid laws and ordering processes may be too cumbersome and constraining to permit responders to get what they need. Governments often lack standing contracts and agreements for specialized resources. Once materials are obtained, poor property-tracking systems leave response agencies vulnerable to public accountability problems and lawsuits.

Remedies to these problems do exist, but they are not broadly implemented. For example, there are one-stop-shop mechanisms available (such as those of the General Services Administration), but these are neither widely understood nor widely used, and the procedures involved must be pre-arranged. Mutual aid relationships can be an effective conduit for support, but these are often informal and are not centrally coordinated. As a result, a single mutual aid asset may be "counted" by several different agencies as part of their resource bases. The wildland fire community uses a very effective nationwide resource ordering and deployment system, but this approach has not been replicated by other disciplines. Moreover, common terminology and standard resource typing are required for such a system to work; these do not yet exist across response disciplines.

Volunteers and donated resources present a particular challenge to incident management. Tracking systems for these resources are weak, and as a result many assets go underutilized. Many organizations have useful capabilities but do not know how to identify or connect to the incident management system, either because they do not understand ICS or because the command system is so fractured it is hard to navigate. Even emergency response agencies that do understand ICS often "self-dispatch" to the scene without coordinating their response. These agencies have important skills, but often deploy without the ability to support themselves with food, water, fuel, shelter, or communications. Also, it is hard to verify the credentials of personnel who show up to help; some are highly qualified, while others have no business being at an emergency scene. Yet there is no easy, standard way to confirm the background and affiliation of volunteers. Likewise, maintaining accountability and tracking volunteer status is equally difficult. As a result, well-meaning volunteers add a significant management burden to already over-taxed incident managers. As noted in Arlington County's report after 9/11, "Organizations, response units, and individuals proceeding on their own initiative directly to an incident site, without the knowledge and permission of the host jurisdiction and the Incident Commander, complicate the exercise of command, increase the risks faced by bona fide responders, and exacerbate the challenge of accountability." ⁴ On top of these problems, much of the material sent to the scene is not useful, but must still be managed - transported, stored, and disposed of. Agencies often lack plans for getting rid of stuff they receive but do not need.

Poor Public Relations

Responders told us they believe that the general public wants instructions about what to do, but that people may not receive or understand the directions government agencies give them. In part, responders say, this is because governments rely heavily on mainstream media. Many people don't pay attention to mainstream media, and therefore don't get the information governments want them to have. Even people who do get the information may not understand the message correctly, especially when the government gives short shrift to pre-incident public education. This problem is exacerbated in the heat of an incident - when agencies fail to use a common message, do not control the message carefully, the pressure to get information out quickly undermines accuracy, and rumors propagate unchecked.

Even when directions are clear, received, and understood, some people do not have the wherewithal to follow them. As our incident managers acknowledged, some people just do not have the will to do as they are told. In the incident managers' view, the public is generally complacent about preparedness. This is borne out by anecdotal evidence. For example, during the recent commemoration of the 1906 earthquake, National Public Radio reported, "Scientists agree that it's very likely another big earthquake will hit the San Francisco Bay area in the next thirty years, but...many people in the Bay Area still live in denial" (April 18, 2006). Interviews with a number of citizens illustrated their point. Few had serious plans or any supplies to sustain them in the event of a major disaster. The lack of wherewithal or will on the part of the public presents a recurring challenge to governments that have not invested enough resources in emergency transportation and shelter.

Findings: Why Don't We Learn?

These lessons relate to some of the most important and involved functions of incident management, so it is no surprise that problems are identified repeatedly in the areas of command, communications, planning, resource management, and public relations. Likewise, responders are most likely to notice concerns in these areas by dint of the effort expended on them during any incident. Moreover, large, complex incidents are inherently challenging to manage. Destructive and unpredictable, they impose extraordinary demands on the decision-making and service-delivery systems of the affected communities. Nevertheless, responders claim that many problems encountered repeatedly are solved anew each time, suggesting that it should be possible to inculcate improvements across time and agencies. It should be possible to solve at least some of these problems once and for all, rather than time and again. This section reports findings that illuminate the challenges to this proposition in five general areas: motivation, reporting, learning, exercising, and resources.

Motivation for Change

Learning is, at its core, a process of growth; thus a successful learning process requires a commitment to change. ⁵ Organizational change is notoriously difficult, but particular challenges attend change in the emergency response arena. One challenge is political traction. Individual citizens rarely see their emergency response systems in action. They generally assume the systems will work well when called upon. Moreover, citizens underestimate the likelihood that disaster will befall them. Yet citizens are confronted every day by other problems they want government to fix - failing schools, blighted communities, and high fuel prices. Politicians tend to respond to these more immediately pressing demands, deferring investments in emergency preparedness until a major event re-awakens public concern. As one incident commander put it, "Change decisions are driven by politics and scrutiny, not rational analysis." High-profile events and the media attention they garner generate opportunities to make changes because public fear prompts politicians to support improvements.

Scrutiny can free up resources for change, but the results can be perverse as well. Until Hurricane Katrina struck, the most momentous event in the public's memory was the 9/11 attack. On the basis of that incident, the president and Congress initiated a major new homeland security policy agenda, including one of the most significant government reorganizations in history. Many of the policies and programs promulgated under the auspices of homeland security are targeted at Weapons of Mass Destruction (WMD) and terrorism. This frustrates emergency responders who continue to struggle to maintain and upgrade their capacity to cope with a myriad of other (more common) threats and hazards. Our experts' frustration on this point was palpable. They find the WMD focus distracting. As one fire chief raged, "It's terrorism, terrorism, terrorism...and I can't use my resources for the things I know I'll face. So how many major non-terrorism incidents do we have to have before DHS get us resources for other things than WMD?" A police chief agreed: "Local agencies are having terrorism shoved down their throats. They can hardly do basic tactical training because of all the mandates for certifications, much less terrorism training."

Even following a major event, it is hard to sustain a commitment to change long enough to accomplish it. After an incident, it takes time to conduct an analysis and identify lessons. Washington D.C. and the public have very short time horizons; neither waits for these reports to move ahead. The government tends to focus on fast (and inexpensive) solutions - quick wins they can point to before public attention wanes. This kind of nearsightedness is inconsistent with meaningful change. By the time reports come out, there is no will (nor funding) to implement changes. By then, leadership has either turned over or moved on to something else. One local manager told us "We thought we did a lot of work with our politicians after the last major incident. But they have better things to do. Five minutes after that incident is over, they're on to something else. There are a lot of gains to be made if they do well [managing a disaster], but an incident is a political flash in the pan for them."

Even in cases where important lessons do make it to the public agenda, the disparate emergency response community lacks a shared vision of what to do about those lessons. Response professionals see desired outcomes differently based on what agencies and disciplines they represent. Our focus group claims this problem has gotten harder since 9/11, because federal involvement in trying to solve problems is so much greater. The federal government has many resources to devote to policy and planning relative to local governments, who do not have much capacity. On the other hand, it is the locals who deliver services, are closest to the needs of the community, and best understand how to meet those needs, whereas federal agencies are removed from the exigencies of emergency response operations. As a result, federal and local agencies talk past each other. Even in cases where federal and local policymakers see problems the same way, federal ambition outstrips local capacity; federal agencies do extensive planning, but there are not enough local resources to meet the federal vision.

Another impediment to change is the episodic nature of significant events. Any given agency experiences incidents fairly infrequently, but looking at the nation as a whole, relevant events occur all the time. For the nation to improve response overall, the emergency response community has to be able to learn from all of these events. This calls for organizations to think of their experiences collectively, and be willing to learn from each other. But it can be difficult for agencies to perceive the experience of others as relevant to their own responsibilities and operations, and it can be hard to prioritize these lessons over the daily problems an agency confronts in its own jurisdiction. One chief told us, "There [are] no teeth in lessons from someone else's experience. We don't really learn from others unless we can really imagine ourselves in that other person's circumstance." Another explained the problem this way:

We fail to recognize and apply the lessons on a daily basis. We kill firefighters over and over again the same way, report after report after report. We look for big lessons, but fail to identify the small ones and apply them in 'onesies' and 'twosies' every day. Why? We don't see the relevance, or think it won't happen here, or we're too parochial. We're tone-deaf to things that happen to other organizations. Distance in time and space makes this worse. As we get further from each other and the event gets further in the past, it is easy to ignore it. And, even if you accept that something needs to be done, how do you manage it with everything else that's coming at you as a priority every day? Small lessons just don't take priority.

Beyond this, it seems that pressure for change from within the discipline does not have the same force as external scrutiny. As we have said, public fear can motivate rapid behavioral change. An example that members of the focus group pointed to was the transformation in the active-shooter doctrine that resulted from the Columbine High School shooting. The protocol changed dramatically in eighteen months, and the change was universal - all S.W.A.T. teams in the country updated their procedures.

Why did such a sweeping change happen so fast on the basis of a single incident? There were three reasons. First, public scrutiny; every community had a school full of kids whose parents were afraid. Second, willingness to admit to an important lesson; the Columbine Police Department stood up and said, "We didn't handle this right." Third, rapid dissemination; the new protocol was sent to every department across the country through the law enforcement information network. Our focus group participants contrasted this example with the fact that it took five years and a lot of firefighter deaths to get Rapid Intervention Teams (RIT) established, even though many in the field knew they were necessary.

All of this suggests that thinking about learning and change in a single agency or discipline faces substantial barriers. Doing this work collectively is even harder, especially when long-standing animosity gets in the way. It is common for agencies to compete for attention and resources daily, and this only gets worse with big incidents. But one chief acknowledged that collaboration is a key enabler of learning. As he said, "If you're not alone in this game it's a lot easier. If you're learning alongside others who face the same problems and will be your partners in a major incident, then you're more likely to obtain broader, more persistent change because you change expectations across organizations."

Review and Reporting Process

Assuming that an agency is open to learning and change, the learning process can be thought of as beginning with the identification of lessons. This is typically accomplished through the publication of After Action Reports (AARs). Our response experts told us that while some reports are very comprehensive and useful, lessons reporting processes are, on the whole, ad hoc. There is no universally accepted approach to the development or content of reports. Moreover, there are often several reports that come out of any given incident. Sometimes joint reports are prepared, but more often agencies or disciplines write their own without consulting each other. These reports differ and even conflict, since perspectives and experiences (even from very reliable sources) vary dramatically, so that "sometimes you wonder if people were on the same incident." It is difficult for an agency seeking to learn from the reports to de-conflict them, since there is no independent validation mechanism to establish whether findings and lessons are "right."

Worse than conflicts and possible inaccuracies, concern about attribution and retribution is a severe constraint on candor in lessons reporting. It is politically dangerous for an agency or a leader to own up to mistakes and problems for fear that the leader or agency will be penalized. To contend with this, lessons are often reported in a much redacted way; as a result, the level of detail required to make a lesson meaningful and actionable is lost. Meaning is also diluted by the lack of a common terminology. The same functions are described using different terms in different disciplines and parts of the country. Or the same terms are used to describe different things. This leads to a lack of understanding, or a false sense of understanding.

Another substantive problem is that the focus of reporting is unbalanced. AARs typically focus on what went wrong, but chiefs want to know what they can do that is right. Reports tend to detail things that didn't work, without necessarily proposing solutions. Incident managers seek a lessons-learned system that provides good answers, solutions, and best practices. They want to hear what to do, instead of what not to do. They would also like to hear about "near misses," things that almost went wrong and could go wrong again elsewhere without preventive action. This kind of reporting requires an additional analytical step; to produce reports that meet these needs, those preparing the reports need to understand not only what happened, but also why it happened and what corrective action would have improved the circumstances. Reports of this depth and quality are relatively rare. Beyond this, many opportunities to learn smaller but valuable lessons are foregone because formal reports are typically only generated for major events, not for small day-to-day incidents. These "less significant" lessons, if disseminated, offer important opportunities to make more manageable developmental changes in response procedures, but there is no mechanism by which these smaller lessons can be easily reported and widely shared.

The value of even well-crafted reports is often undermined because they are not distributed effectively. Most dissemination is informal, and as a result development and adoption of new practices is haphazard. Generally, responders must actively seek reports in order to obtain them. Lessons do get reported at conferences, but these discussions rarely trickle down to the front line. There is no trusted, accessible facility or institution that provides lessons learned information to first responders broadly, although some disciplines do have lessons repositories. (The Wildland Fire Lessons Learned Center and the Center for Army Lessons Learned are two prominent examples.) And there are some consolidated collections of reports that attempt to fill this need. (See, for example, the Memorial Institute for the Prevention of Terrorism's Lessons Learned Information Sharing web site. ⁶ )

Learning and Teaching

Failure to learn is due, in part, to a lack of systems to identify and disseminate lessons. Even when lessons are identified, our response experts told us, most learning and change processes lack a formal, rigorous, systematic methodology. Simplistically, the lesson learning and change process iterates through the following steps: Identify the lesson ? recognize the causal process ? devise a new operational process ? practice the new process ? embed/institutionalize and sustain the new process. ⁷ It is apparent in practice that there are weaknesses at each of these steps.

Learning begins with an analysis to identify the causal process that underlies the lesson. Absent this, there can be no confidence that a remedy will work and this kind of analysis is rare. One manager explained, "We don't study lessons carefully enough and apply them in a serious way. We don't drill down into the details of what changes are really required to address lessons." In particular, our incident managers told us that agencies find it difficult to think in general terms to be able to see how lessons from one incident or discipline might apply to another. This dilemma is intensified by the fact that the emergency response disciplines writ large lack a common operating doctrine. Without common, accepted conventions against which to compare behavior, it is hard to spot deviations and inconsistencies that suggest the need for learning and change. Moreover, agencies tend to consider individual incidents and particular lessons in isolation, rather than as systems or broad patterns of behavior. A chief put it this way: "We don't look enough at the relationships of components. Day to day, we focus too narrowly and short-term, so our problem-solving approach doesn't consider the whole system."

Following the analysis through which lessons are identified and appropriate remedies understood, practice is required to inculcate new behavior. Often the work of identifying relevant lessons and devising corrective actions makes the agency feel it now knows what to do. One responder admitted, "We spend a lot of time writing AAR's, which gives us the sense that we learned lessons, but the lessons are not consolidated into a training regimen, and so we don't actually learn them." Our focus group participants agreed that practice gets shortchanged. As a result, the link between the last two steps in the learning process seems especially tenuous. Agencies that do get to the point of practicing a new process are lulled into a false sense that they have now corrected the problem. But when another stressful event happens, it turns out this new process is not as firmly embedded as the agency thought. "We feel committed to new courses of action, but then they fall apart on exercises, much less incidents." A lack of practice means that processes have not been rehearsed well enough to work out details and problems, or to develop trust in the new process. Since responders do not really understand and trust the new process, they revert to their old familiar ways. These old habits seem "safer," even though past experience has shown they do not work. One chief described this pathology as follows: "Lessons represent dramatic changes that are hard to entrench. So we fail because we're not disciplined enough, and we fall back to old habits rather than sticking to what we learned."

This problem is rarely noticed until another event occurs because follow-up is inadequate. Our experts told us that feedback after implementing new practices is typically informal and passive; it comes from simply noticing improvement, rather than actively testing for it. One responder summed up the problem: "There are breaks throughout the cycle. Even if we could identify lessons, identify corrective actions, implement them, train them, and exercise them, how do we know if the changes solved the problem?" Lessons are not clearly linked to corrective actions, then to training objectives, then to performance metrics, so it is difficult for organizations to notice that they have not really learned until the next incident hits and they get surprised. As an Oklahoma City Chief told us about his experience after the Murrah Building bombing, "We'd been through a major incident, and when the next one was inbound, we thought we were ready, thought we'd learned and knew what to do. Then the incident hits and we have the same problems all over again. Turned out we didn't really learn what we thought we had." Responders from New Orleans echoed this sentiment: "We did Mardi Gras so well that Katrina felt like a sucker punch."

According to our focus group experts, fixing the weak links in the lessons learning cycle requires that response agencies have a deeper understanding of how to learn. But, they say, the learning process is not taught in our emergency response educational institutions. For example, the National Fire Academy (our premiere institution of learning in the fire service) doesn't teach learning science or systems thinking at any level of sophistication. Our emergency response agency leaders especially need this knowledge. One manager highlighted this need: "We don't know if we are even creating the mechanism appropriate to learning in our agencies."

Exercising

Perhaps the key mechanism for testing, practicing, refining, and inculcating new lessons-derived behaviors is exercising. Almost every AAR discusses the crucial role that training and exercising play in building capacity. Unfortunately, our AAR review and focus group betrayed several important weaknesses in the way disaster exercises are designed and executed.

One design problem is striking the right balance between the known and the unknown. On the one hand, it is very important to prepare for what is likely. But preparing for the events that will probably occur does not get responders ready for the unusual, unexpected, and unforeseen, and how to handle these kinds of circumstances. One chief asserted that "we don't use our imagination in preparedness," but also acknowledged the opposite problem: some exercise scenarios are so far-fetched that they are a waste of time. At the same time, responders, like citizens, would rather believe that the possible worst case just will not happen. This kind of denial stems in part from the fact that it is important for emergency responders to have confidence and courage in the face of extreme adversity. One incident manager explained: "The hardest thing is to train a firefighter or a cop to know that they're overwhelmed. They are trained to feel like they can handle it. It's a rude awakening to recognize your own mortality, but we've got to."

Another design problem is a lack of realism, not with respect to scenarios, but in terms of what is required of responders and incident managers. Exercise procedures are typically simplified, compared to how they would actually unfold in a real event, in order to meet the time and resource limitations of exercises. This masks complexity, however, and responders often fail to appreciate what it really takes to get work done; when reality hits, "the devil is in the details." One senior leader explained, "In planning and exercising, everything works nicely, but this doesn't happen in the real world." This lack of realism sometimes stems from the fact that exercises are rarely held without advance notice and tend to escalate progressively, rather than erupting suddenly on a broad scale. Sudden-onset incidents present a particular challenge for which responders need better preparation. One expert described his concern this way:

As incidents escalate progressively in front of us, we build and add systems. We do this naturally. But what happens when the incident starts out big and requires a large system right out of the gate? We don't know how to handle instantaneously large, complex incidents. We go in thinking "I can handle this." We don't go in thinking "I'm overwhelmed from the get-go."

Finally, a major impediment to exercising is fear of failure - not in reality, but in the exercise. Our response experts told us that most exercises have a punitive tone. Exercise designers and evaluators have some "school solution" in mind that is never revealed to exercise participants. The participants do not understand in advance what the expectations are for success and the exercise objectives are unclear. As a result, participants make obscure decisions during the exercise, for which they are later criticized. This has occurred often enough that responders are reticent about participating in exercises. One senior manager explained the problem this way:

We exercise wrong. People don't come to exercises because they're afraid they'll be tested, that they'll make mistakes, and that they'll be embarrassed. They make strange decisions that cause the exercise to veer off. We don't train people how to operate first and then test them afterwards. Instead, we throw them in blind, and then tear them apart afterwards. We just expect them to make the decision that we had anticipated and then criticize them when they don't.

This situation is not helped by the fact that exercises are often designed and delivered by contractors, whose interests may have more to do with business than with improving response performance in the field during a major incident. Our focus group participants fear that this punitive approach will only get worse if exercises become DHS's main tool for assessment and funding allocation.

There are also important imbalances in exercise goals and the content of exercise scenarios. Responders told us that the preoccupation with terrorism and WMD means that DHS mandates state and local governments to perform exercises they do not need, dealing with scenarios that are far down the list of likely events or do not focus on important capability gaps. In fact, a narrow focus on WMD is potentially damaging to our preparedness. There is only so much room on an agency's training schedule. If WMD displaces too much, there is not enough time to focus on training to address lessons the field has identified. Even making room for the most pressing lessons is already hard. Our experts assert that state and local governments need to identify their greatest threat and weakest capability and define their own relevant exercise needs.

Another common problem is that exercises have proliferated in light of events and studies highlighting weaknesses in preparedness. Not only are there too many exercises, there are too many goals for each one. As a result, locals are inundated with exercise requirements to the point that their participation is not productive. Our experts suggest fewer exercises with narrower and sharper objectives: "We are already exercised to death. More exercises is not the answer." Finally, exercises fail to target one of the most important levers of preparedness: regional relationships. Agencies either exercise in isolation with simulated interactions, or the interactions that are required by an exercise do not mimic those that would operate in a real disaster. As a result, agencies fail to derive perhaps the most important benefit of the exercise process: relationships with other agencies, jurisdictions, and disciplines.

Execution problems further undermine the value of exercises. A major problem is that many responders miss exercises, because they happen on a particular day and shift, and exercises of a particular type may happen only one time or once a year. So only some of the force gets the experience. Moreover, while the same agencies may participate together in several exercises, the participants vary from exercise to exercise. As a result, people have a "one-shot" experience and do not get a chance to learn from their mistakes and then try it again. Worse, uneven participation means that agencies miss the opportunity to build strong, trusting relationships. Finally, an important result of exercises is to expose deficiencies so that they can be examined and corrected, but fear of retribution or penalties impede honest reporting. Too often, public officials report wonderful successes and do not reveal problems. As a result, the public thinks everything is under control and has unrealistic expectations when a bad incident actually happens and agencies do not get the resources they need to correct problems.

Resource Constraints

Commitment to learning is wasted if resources are not available to support the process. Unfortunately, funds available to sustain corrective action, training, and exercise programs are even leaner than those available for staff and equipment. One chief asked, "Even if we read every AAR, where do we get capacity to implement lessons in our organizations?" Part of the problem is that investments in basic capacity are not as marketable - as "sexy" - as equipment for combating terrorism. Furthermore, carefully conceived, sophisticated, progressive training and exercise programs are very expensive in time and dollars. Unlike the military, which spends a great deal of time training when they are not actually fighting, emergency responders have many other ancillary duties aside from responding to calls. Response agencies cannot pull personnel off the line or off these duties for training and exercises. And, our experts admitted, we shouldn't be investing resources in training and exercises unless we make the effort to improve our lessons-learning processes. "If you don't get the lessons right," they say, "you chase hollow solutions and throw money around without actually solving problems."

To synthesize our findings, we believe the fundamental challenge is that it takes long-term resource commitment and organizational discipline to solve recurring problems. Too often, however, political support is too transient as other, more visible, concerns divert resources from longer-term preparedness activities. Even in the emergency response domain, attention devoted to terrorism distracts from developing the basic capacity needed to respond to more common incidents. Agencies are easily distracted by their daily missions, as well. Because lessons from major incidents are not easily accessible, are not detailed enough to be useful, and their relevance is not immediately obvious, agencies are reticent about committing the time and effort needed to really understand, develop, and implement corrective actions that would improve their performance. Even if they do decide to adopt new procedures, inadequate practice prevents transitional changes from taking hold. Further, those changes most likely to become embedded are smaller internal adjustments, rather than broad culture changes that arise from a vision and doctrine shared across agencies, jurisdictions, and disciplines.

Recommendations

We asked our participants for ideas to improve our ability to learn the lessons of the past. Their ideas centered around three themes.

1. The need to radically improve the way we train and exercise.

2. The need for a comprehensive, nation-wide capability to gather and validate the information we learn from incidents, develop and vet corrective actions, and disseminate them to those who must inculcate the changes.

3. The need for incentives to institutionalize lessons-learning processes at all levels of government.

Training and Exercising

Across the board, our study participants felt the key to learning lessons is to improve the way we train and exercise. Most importantly, exercises must be recast as learning activities targeted at improving performance, not as punitive tests where failure is perceived as threatening an organization's ability to garner funding or maintain political favor. This requires improvements in exercise planning. In particular, exercise planners should explicitly link the lessons a jurisdiction seeks to learn to a limited set of focused exercise objectives, rather than trying to accomplish numerous ill-defined goals. Our participants recommended that exercise planning should follow the military-style crawl à walk à run structure where participants first learn expectations about appropriate actions and decisions (crawl), then move slowly through a scenario taking the time to practice decision making (walk), and then pick up the pace and the challenges as they get more adept (run). This suggests that responders should engage in smaller, more frequent, narrowly tailored exercises with limited goals before they get to exercises on the scale of TOPOFF. An exercise program with this sophisticated, progressive structure is resource-intensive and will require federal support. DHS should reorient and enhance their exercise planning resources to help state and local agencies plan and execute exercises that fulfill these goals.

The participants offered advise for their own agencies as well, beginning with making training for large-scale events tougher. While the table-top exercises commonly employed provide valuable practice with strategic and tactical decision making, they are not realistic enough to test jurisdictions' abilities to coordinate resources and communicate with each other. We must find a way to introduce the chaos and common failures likely in a real event into state, regional, and local exercises. And local agencies need to find a way to inculcate practice with the protocols and behaviors suggested by lessons in daily activities. For example, jurisdictions that explicitly set up incident commands for every event they respond to - even the small ones - are better able to inculcate this way of operating as a habit responders will draw on during major incidents when it is especially necessary.

Finally, individual agencies also need to do a better job of adopting disciplined processes for reporting lessons and updating plans to reflect them, so that lessons, planning, training, and exercising can be better integrated. None of the lessons identified in this study as recurring are hazard-specific. They arise from incidents of all types. This suggests that even though disasters seem unique, solutions to them can be generalized. Most experienced incident managers will tell you that the best way to make a decision in an emergency is to make it well ahead of time, before an incident happens and before the specific nature of the problems that will arise is known. This suggests two things: first, that an all-hazards perspective is appropriate and should be emphasized, and second, that many of the issues that will arise from an incident of any type can be addressed in a rigorous planning process. To support lessons-focused planning, agencies must establish and follow a requirement to document events and lessons learned immediately post-operation, for small events as well as large incidents. Adopting a standard format for this process will make it easier for responders to follow this mandate. Agencies must also require that lessons learned (both theirs and other organizations') are consulted and considered as plans are revised. To assist this process, the incident managers in our study see an important role for the federal government in creating a national capability to identify and advocate lessons learned.

National Emergency Response Lessons Learned Institute

Learning lessons depends on the development of a robust analytical capability. Such a capability could be the core of a new national doctrinal institute or part of an existing federal academy or preparedness activity. Any of these arrangements would be a significant undertaking, however, and would not succeed as simply an "additional duty" of an existing organization. Analysts who support this institution would need to be able to understand the information in AAR's, translate them into a common language, identify and resolve conflicting information or conclusions, develop the required changes in policy and procedure, vet or test those new concepts, and deliver them to the organizations who need them. This would require analysts with both expertise in learning science and in-depth knowledge and experience in the emergency response and incident management domains.

For this institute (whatever its configuration) to be effective, it must begin by promulgating a sound reporting system. Good reporting is demanding and resource-intensive; it requires clear criteria for what is to be reported, a standardized reporting process, a robust and secure data management capacity, and a user-friendly interface. A universal national reporting methodology would help response organizations understand what they should include in their reports so that they will be useful tools for learning and change. A common format would make it easier for agencies to understand each other's reports.

Once reports are submitted, the institute's analysts could work on understanding the lessons indicated in reports in order to consolidate findings and develop proposed procedural changes and alternatives, additions to doctrine, or new concepts of operations. Then, the institute should use sophisticated information technology to make their findings easily available across response disciplines. Tools such as databases with smart search engines, electronic update bulletins, and web-based training, should be employed so that responders could easily find information relevant to their missions, disciplines, service responsibilities, and hazard environments. Just creating a database will not be enough, however. It must be broadly recognized across response disciplines as a definitive, comprehensive, and valid information source. To ensure the ideas, information, and proposals generated by the institute are trusted by the response community, a peer review process to validate the lessons and ensure the proposed solutions are legal, workable, and effective is essential.

Also essential are activities to push important information out to users proactively and regularly through several modes and in a variety of forums. As examples, safety bulletins and time-sensitive alerts should be rapidly disseminated. A system akin to the National Law Enforcement Telecommunication System (NLETS) and other messaging systems currently used by police agencies could be effective. The institute should also have a regular publication and submit articles and columns in established emergency response trade journals. The institute should sponsor broad, multi-disciplinary, lessons-focused conferences. Finally, the institute should conduct senior leadership training forums that focus not only on lessons that need to be learned but on how to learn them.

A crucial concern when developing an effective lessons analysis institute is liability. For a lessons reporting and dissemination system to have integrity requires that those who report be protected from retribution. Absent this protection, reports cannot be specific enough to be useful. If lessons databases are fodder for lawsuits, jurisdiction attorneys may prohibit participation. The need for a high level of protection argues for enactment of such a system in legislation that assures immunity from Freedom of Information Act requests and restricts the use of reports in lawsuits, regulatory enforcement, or personnel actions. There are models for how a system like this could work. For example, the Aviation Safety Reporting System (ASRS) is a joint NASA-FAA initiative that collects, analyzes, and responds to voluntarily submitted aviation safety incident reports in order to lessen the likelihood of aviation accidents. Pilots, air traffic controllers, flight attendants, mechanics, ground personnel, and others involved in aviation operations submit reports to the ASRS when they are involved in, or observe, an incident or situation in which aviation safety was compromised. All submissions are voluntary and reports sent to the ASRS are confidential. The FAA has committed not to use ASRS information against reporters in enforcement actions. ⁸ Another example that demonstrates how this might be developed for the emergency response community is the National Fire Fighter Near-Miss Reporting System, which is intended to be a voluntary, confidential, non-punitive, and secure reporting system targeted at improving firefighter safety by sharing the details of unintentional unsafe occurrences. It is currently funded by grants from the Department of Homeland Security's Firefighters Grant Program and the Fireman's Fund Insurance Company. ⁹

Incentives

The incident commanders who participated in our study pointed out that discussions of lessons are moot unless they can be disseminated to the grass-roots level nationwide so that line responders can adopt them. They noted that it is important to recognize that people respond to incentives - that is, line troops will change their behavior to reap rewards or avoid punishment. Currently, fear of retribution drives responders away from participating in exercises or reporting their mistakes, for example. We need to develop incentives that cause regions and localities to support and promote lessons reporting, change, and learning. One obvious approach is to make federal and state funding contingent on developing and using lessons-learning systems. This kind of "carrot" can work. For instance, Fire Act and SAFER grants require that agencies participate in the National Fire Incident Reporting System for the grant year and three subsequent years. This is one possible model.

In order to achieve broad consistency, our participants argue that it is important to promote a regional commitment to identifying problems and adopting best practices. They believe that local agencies were "more likely to adopt and indoctrinate lessons if the people they work with every day across a region are doing the same thing." Identifying appropriate regions is not a trivial task, however. Who constitutes the "right" set of local collaborators depends on the characteristics of the prospective participant jurisdictions, the resources they have available, and the hazards they face. Successful regions cannot be imposed arbitrarily; they should be formed according to some commonality identified locally. This will be especially challenging when natural regions cross state lines. Regionalization could be enabled by enhancing (and in some cases fixing) existing regional processes and mechanisms (e.g. Local Emergency Planning Committees and State Authorizing Authorities). Regional response plans could be a prerequisite for receiving funds from a regionally-targeted preparedness grant program. A federally supported regional exercise program could also promote broader inter-jurisdictional work.

Conclusions

The fact that challenges to learning lessons persist, despite regular experience with them, is a serious concern. In today's environment, where the emergency response mission space is expanding dramatically to include broader homeland security responsibilities, the ability to capitalize on experience and improve capacity is ever more important. But organizations cannot just be told to "change." Enduring change needs to address the structure, system, and culture of an organization so that patterns of behavior can be adjusted. Truly institutionalizing a new process requires long-term commitment. This is what makes learning processes especially vulnerable: there are too many short-term distracters. Other political priorities, sensational concerns like terrorism, workforce turnover, other concurrent organizational change efforts, and daily missions all conspire to derail organizational transition. As a practical matter, then, the main problem with lesson learning can be seen as a lack of will and commitment, rather than a lack of ability. If lessons learned become a priority for leaders - especially local leaders who will be called to manage disasters directly - then lessons learned have a better chance of becoming a priority for everybody. Moreover, this commitment needs to be vertical; federal agencies must also commit to identifying and learning the lessons that are relevant to them. As one responder put it, "You can fix all the wagons locally, but if the wheels fall off FEMA's wagon, the system fails."

An additional conclusion is that most big lessons are inter-agency lessons. Learning them requires learning within and across agencies. It is not enough for agencies to try to learn these kinds of lessons in isolation. Despite its profound advantages, federalism gets in our way: we have national, state, and local governments but few robust regional forums for decision-making. Our system lacks substantial support and incentives for regional (multi-state within the nation, or multi-local within and across states) activities and for broad integration across the response disciplines. Disasters are regional - they do not recognize jurisdictional boundaries or disciplinary parochialism. Our systems for learning from disasters must therefore span these barriers.

Another key observation is that, as one reviewer of this article pointed out to us, much of what after action reports focus on is tactical, operational, and retrospective in nature. Reports tend to offer relatively little insight into the more strategic dimensions of disasters, and do not tend to take a prospective view of what can be done to prevent them. Our purpose in this article has been to consider how well we learn from our experiences, with the contention that it is by learning from successes and mistakes that we can be better prepared to act when something else happens. Yet, we also believe that prevention is an important (and often ignored) goal. This suggests two things: first, those who prepare reports often forego the opportunity to comment on how to avoid problems altogether (rather than contending with consequences); and second, further research is warranted to understand learning in the context of prevention (as opposed to response).

Finally, from an academic perspective, focused research can improve our understanding of how to make lesson learning work well. Immediate research opportunities include more rigorous textual and content analysis of the AAR's to validate our suggestive findings, to identify causal processes (understanding of which may enhance learning), and to understand the differences in perspectives that emerge across all agencies participating in the same incident. Researchers should bring the learning science and social psychology literature to bear on developing approaches to effective learning tailored to the challenges of preparedness and emergency response. In short, helping organizations navigate the complexities of lessons learning should be informed by the substantial academic literature that has developed around this issue. A brief synopsis of some key insights is included in Appendix C for reference.

The very real consequence of failing to learn lessons is loss of lives and property. In short, as one responder told us: "If we don't learn these lessons, people are going to die again, because we failed to fix the problems that killed people the last time." We should not belittle the magnitude of this challenge, however; problems recur because they are inherently very difficult to solve. If solutions were evident, emergency response professionals would have adopted them long ago. This should motivate agencies in all emergency response disciplines and at all levels of government to give serious attention to the goal of inculcating a culture of learning from past disasters to prevent future losses.

This paper is based on research sponsored by the Memorial Institute for the Prevention of Terrorism and the Department of Homeland Security Science and Technology Directorate's Emergency Preparedness and Response Portfolio. We are grateful to the incident managers who participated in this project, to Michelle Royal for her extensive research support, and to Tempril Moore for her administrative support.

Amy K. Donahue is associate professor of Public Policy at the University of Connecticut. Her research focuses on the performance of emergency services agencies and citizen preferences for public safety. Since 2002, Dr. Donahue has served as a technical advisor to the Office and Department of Homeland Security, helping develop programs to meet the technology needs of emergency responders. From 2002-2004, Dr. Donahue served as Senior Advisor to the NASA administrator, and she now sits on NASA's Aerospace Safety Advisory Panel. Dr. Donahue has many years of training and field experience as an emergency responder, including managing a 911 communications center, and working as a firefighter and medic. She served in the U.S. Army, rising to the rank of Captain. Dr. Donahue holds her Ph.D. in Public Administration and her M.P.A. from the Maxwell School of Citizenship and Public Affairs at Syracuse University. Her B.A. from Princeton University is in Geological and Geophysical Sciences.

Robert V. Tuohy is vice president for Strategic Planning at Hicks & Associates, Inc. where he focuses on Homeland Security policy and has led Project Responder for four years. Besides leading the development of Project Responder's National Technology Plan for Responding to Catastrophic Terrorism, he is also assisting the Department of Homeland Security in assessing emergency response preparedness and addressing the nation's Border & Transportation Security technology needs. Prior to joining Hicks & Associates, Mr. Tuohy served as the director of Science and Technology Plans & Programs in the Office of the Secretary of Defense where he was responsible for developing and coordinating the Department's science and technology strategic planning and program assessment activities. Mr. Tuohy has a B.A. in Applied Behavioral Sciences from the National-Louis University, and an M.S. in Science and Technology Commercialization from the University of Texas at Austin.

Appendix A: After Action Reports Reviewed

Allan, Susan. 2002. "Anthrax - Local Response and Lessons Learned." Presentation to the NACCHO Annual Conference by the Health Director, Arlington, VA.

Arlington County. 2002. Arlington County After Action Report On The Response To The September 11 Terrorist Attack On The Pentagon. Prepared for Arlington County by Titan Systems Corporation under a grant from the Department of Justice, Office of Justice Programs, Office for Domestic Preparedness, under Contract Number GS10F0084K, Order Number 2001F_341.

Crupi, Robert S., Deborah S. Asnis, Christopher C. Lee, Thomas Santucci, Mark J. Marino, and Bruce J. Flanz. 2003. "Meeting the Challenge of Bioterrorism: Lessons Learned from West Nile Virus and Anthrax." American Journal of Emergency Medicine 21 (1): 77-79.

Ekard, Walt, W. Harold Tuck, Jr., and Deborah Steffen. 2003. San Diego County After Action Report: Firestorms 2003.

Federal Emergency Management Agency. 2003. Columbia Recovery Mission Hotwash Report.

Florida Governor's Disaster Planning and Response Review Committee. 1993. Final Report.

Gursky, Elin, Thomas V. Inglesby, and Tara O'Toole. 2003. "Anthrax 2001: Observations on the Medical and Public Health Response." Biosecurity and Bioterrorism: Biodefense Strategy, Practice, and Science 1(2): 97-110.

Herbert, W. Robert Chairman, Claire A. Collins, Member, Wm. B. Rowland, Jr. 2003. An Assessment: Virginia's Response to Hurricane Isabel Submitted to The Honorable Mark R. Warner, Governor of Virginia. Arlington, VA: System Planning Corporation.

Jenkins, Brian Michael and Frances Edwards-Winslow. 2003. Saving City Lifelines: Lessons Learned in the 9-11 Terrorist Attacks. San Jose State University: Mineta Transportation Institute.

Manzi, Catherine, Michael J. Powers, and Kristina Zetterlund. 2002. "Critical Information Flows in the Alfred P. Murrah Building Bombing: A Case Study." In Terrorism Studies Series, Special Report 3 prepared for the Memorial Institute for the Prevention of Terrorism.

Memorial Institute for the Prevention of Terrorism. 2002. Oklahoma City - Seven Years Later: Lessons for Other Communities.

Murphy, Gerard P. and Chuck Wexler. 2004. Managing a Multijurisdictional Case: Identifying the Lessons Learned from the Sniper Investigation. Police Executive Research Forum for the Office of Justice Programs, U.S. Department of Justice.

National Aeronautics and Space Administration. 2003. "Interagency Jointness (Initial Lessons Learned from Columbia Shuttle Recovery Operations)." Draft summary from a Presentation to the NASA Advisory Council.

National Commission on Terrorist Attacks. 2004. The 9/11 Commission Report: Final Report of the National Commission on Terrorist Attacks Upon the United States. New York: W. W. Norton & Company.

Oklahoma Department of Civil Emergency Management. 2003. After Action Report: Alfred P. Murrah Federal Building Bombing, 19 April 1995 Oklahoma City, Oklahoma. Department of Central Services Central Printing Division, Publications Clearinghouse of the Oklahoma Department of Libraries.

Oklahoma Department of Civil Emergency Management. 2005. After Action Report: Alfred P. Murrah Federal Building Bombing.

Rothstein, Alcalde, Elster, Majumder, Palmer, Stone, and Hoffman. 2003. Quarantine and Isolation: Lessons Learned from SARS. Centers for Disease Control and Prevention.

Thompson, Kimberly M., Robert E. Armstrong, and Donald F. Thompson. 2005. Bayes, Bugs, and Bioterrorists: Lessons Learned from the Anthrax Attacks. National Defense University Center for Technology and National Security Policy.

United States Fire Administration . 1999. "Wanton Violence at Columbine High School" Technical Report Series USFA-TR-128.

United States Fire Administration, Federal Emergency Management Agency. 2004. Responding to Incidents of National Consequence: Recommendations for America's Fire and Emergency Services Based on the Events of September 11, 2001, and Other Similar Incidents. (FA-282-May 2004).

White House. 2006. The Federal Response to Hurricane Katrina Lessons Learned.

Appendix B: Focus Group Participants

Ed Bailor, Inspector, U.S. Capitol Police (Retired)

Steve Glynn, Chief of Special Operations, New Orleans Fire Department

Sam Gonzales, Chief of Police, Oklahoma City Police Department (Retired)

Jon Hansen, Assistant Chief, Oklahoma City Fire Department (Retired)

Bob Ingram, Chief-in-Charge of Haz-Mat Operations, New York City Fire Department

Kevin MacCurtain, Acting Fire Commissioner and Chief of Department, Boston Fire Department

Tom Miner, Program Manager, Washington Task Force One FEMA Urban Search and Rescue Team

Jon Olney, Battalion Chief, Kern County California Fire Department (Retired) and National Type 1 Incident Management Team Logistics Section Chief

Jim Schwarz, Chief, Arlington County Fire Department and Acting Director, Arlington County Office of Emergency Management

Mark Willow, Deputy Commander Policy and Planning, New Orleans Police Department

Appendix C: Literature On Learning and Change

This paper is about how emergency responders can more ably recognize flaws in past behavior and implement new behaviors that will result in better outcomes in the future. In short, we are concerned with learning and change. These are complex fields of theory and research. A brief review of some established insights from the literature is instructive.

Fiol and Lyles define learning as "the development of insights, knowledge, and associations between past actions, the effectiveness of those actions, and future actions." ¹⁰ Learning is multi-dimensional: it has cognitive, behavioral, and emotional components, each of which is susceptible to particular instructional or educational approaches. ¹¹ The cognitive domain involves the attainment of knowledge and the development of intellectual skill. Rudimentary learning in this domain includes information acquisition and comprehension, while sophisticated learning involves synthesis and evaluation. The affective domain deals with emotional intelligence or the emergence of attitudes, values, and feelings associated with a particular phenomenon. Receiving and responding to these phenomena are at the lowest levels of learning, while organizing and internalizing values associated with the phenomena are at the highest levels. The behavioral domain involves learning that requires physical activity, ranging from simply observing an activity to adapting the skill to new purposes or origination of new action.

It is evident that learning is an individual-level activity - people use their brains to observe, think, reason, and develop new insights. But since the early 1970s it has been understood that learning also occurs at the collective level. That is, organizations can be considered to learn when they acquire, process, store, and distribute knowledge, understanding, techniques, and practices. ¹² Organization theorists have long discussed how learning is shaped by the use of routines to guide behavior. Levitt and March summarize this body of thought as follows: Organizations depend on accepted routines (operational rules, procedures, and conventions that may be formal or informal). That is, they tend to follow habits believed to be appropriate to their circumstances, rather than selecting actions to achieve a particular result. These routines develop incrementally over time as a result of past experience, rather than consciously in anticipation of future conditions. Organizations adjust their routines based on successive comparison between outcome targets and the outcome levels they actually achieve. Thus, historical experience and established routine profoundly shape organizational learning. ¹³ From these insights, Levitt and March draw the important conclusion that "Organizations are seen as learning by encoding inferences from history into routines that guide behavior...The experiential lessons of history are captured by routines in a way that makes the lessons, but not the history, accessible to organizations and organizational members who have not themselves experienced the history." ¹⁴

Organizational behavior (routines) change as errors develop in the organization's collective memory, as new experiences or the experiences of other organizations threaten the veracity of organizational beliefs about what is appropriate, and as the results of organizational activities diverge from desired or expected outcomes. As a result, organizations adjust their behavior. Meyer terms this adjustment "adaptation," and suggests that adaptation takes two forms: deviation-reducing (where adjustments are made based upon existing organizational assumptions) and deviation-amplifying (where adjustments use new causal relationships founded on revised assumptions). ¹⁵ To the extent that organizations must revise their assumptions, learning is fundamentally about changing culture, where an organization's culture is its system of shared values, perceptions of work, and perceptions of success. ¹⁶ To establish culture, assumptions are made and tested. When they are upheld, they become embedded ("frozen," in Lewin's classic conceptualization). ¹⁷ Culture change occurs when assumptions are challenged. Alternatives are proposed and tested, and if they are upheld repeatedly, adjustments may be made to adopt these alternatives. While this process of examining and revising assumptions is tacit, its effects are practical: for new organizational approaches to be upheld, organization members must find them credible and witness short-term "wins." In short, lesson learning can require culture change.

An important question is what kind of change is warranted in light of what has been learned. An appreciation of the change process can help managers understand how to instigate and sustain it. ¹⁸ Ackerman distinguishes three types of change. ¹⁹ The simplest form of change - and a process that organizations commonly and comfortably use - is developmental change, which can be thought of as an incremental process of improvement and increasing sophistication. This form of change involves adjusting, enhancing, or correcting what already exists. It generally targets skills or methods that fall short of expectations or requirements. At the other extreme, the most difficult and disruptive kind of change is transformational change. This is radical change typically prompted by some shock to the organizational system that challenges the organization's assumptions about environmental demands, resulting in a period of chaos from which an entirely new culture or way of doing business emerges.

An intermediate form of change is transitional change, where an organization discards its old ways of accomplishing its work in favor of new processes. ²⁰ Transitional change is planned, defined by movement to a known state. Typically an organization continues to use its existing process, but becomes increasingly dissatisfied with this process, either because it doesn't work well or it becomes known that something else works better. The organization develops a new process, which it then begins to "try out" in parallel with its old process. During this period of overlap, an organization works in its old process but may increasingly dip into the new process. As the organization gets comfortable and confident, it ultimately switches to the new process. But, if it doesn't have long or strong commitment, a difficult event may cause the organization to revert to its old habits.

What do these insights about learning and change suggest about why change in response to lessons identified from disasters is so hard? Developing a sustainable sense of commitment to a new process requires a persistent sense of urgency about change and improvement, but other political exigencies may overwhelm commitment to emergency response. Since disasters are infrequent, agencies do not have reason to adjust their operating assumptions, and lack continuous opportunities to test out and embed new processes. In effect, organizations will stick with their accepted routines, absent persistent challenges to their assumptions or feedback about their performance relative to desired outcomes. This problem is made harder when an agency is asked to consider a new process that comes, not out of its own experience, but from someone else's. Levitt and March tell us that organizations can capture the learning of other organizations through the transfer of encoded experience, though the mechanisms of diffusion (shared learning through a network) are even more complex than at the level of a single organization. ²¹ Thus we can anticipate that, absent an explicit strategy to learn and change, the emergence of improved behaviors may be slowed or even obstructed.

------------

1. David A. Garvin, Learning In Action, A Guide to Putting the Learning Organization to Work (Boston: Harvard Business School Press, 2000): 106-116.

2. Christopher Bellavita, "Olympic Security After Action reports and Why They Are Ignored," After Action Reports (2002).

3. The White House, The Federal Response to Hurricane Katrina Lessons Learned (2006).

4. Arlington County After Action Report On The Response To 9 11 Terrorist Attack On The Pentagon (2002), 12.

5. The relationship between learning and change is well established in the organizational learning literature. We do not mean to suggest that learning and change are synonymous, but that the purpose of organizational learning in the context of disaster management is changed (presumably improved) behavior. As Argyris and Schön, note, " an important species of organizational learning consists in an organization's improvement of its task performance over time." Chris Argyris and Donald A. Schön, "What is an organization that it may learn?" In Organizational Learning II: Theory, Method, and Practice (Menlo Park, CA: Addison-Wesley Publishing Co, 1996), 4.

6. See Memorial Institute for the Prevention of Terrorism's Lessons Learned Information Sharing web site at https://www.llis.dhs.gov

7. These steps are synthesized from a larger discussion by Argyris and Schön, who explain, in part, that "Organizational learning occurs when individuals within an organization experience a problematic situation and inquire into it on the organization's behalf. They experience a surprising mismatch between expected and actual results of action and respond to that mismatch through a process of thought and further action that leads them to modify their images of organizations or their understandings of organizational phenomena and to restructure their activities so as to bring outcomes and expectations into line, thereby changing organizational theory-in-use. In order to become organizational, the learning that results from organizational inquiry must become embedded in the images of organization held in its' member's minds..." Ibid., 16.

8. See http://asrs.arc.nasa.gov/main.htm for a full explanation of this system.

9. See www.firefighternearmiss.com/home.do

10. C. Marlene Fiol and Marjorie A. Lyles, "Organizational Learning," Academy of Management Review 10 (1985): 803-813.

11. See, in particular, Edgar H. Schein, "How can organizations learn faster? The challenge of entering the green room," Sloan Management Review 34 (1993): 85-93; David R. Krathwol, Benjamin S. Bloom, and Bertram B. Masia, Taxonomy of Educational Objectives Book 2: Affective Domain (New York: Longman, 1999); Benjamin S. Bloom, ed., Taxonomy of Educational Objectives, Handbooks I: Cognitive Domain (Wokingham: Addison-Wesley, 1956).

12. See, for example, Argyris and Schön and Lloyd Baird, John C. Henderson, and Stephanie Watts, "Learning from action: an analysis of the Center for Army Lessons Learned (CALL)," 36 (1997): 385-396.

13. Barbara Levitt and James G. March, "Organizational Learning," Annual Review of Sociology 14 (1988): 319-340.

14. Ibid, p. 320.

15. A. D. Meyer, "Adapting to environmental jolts" Administrative Science Quarterly 27 (1982): 515-537.

16. Argyris and Schön; Also Edgar H. Schein, "Three cultures of management: the key to organizational learning," Sloan Management Review 38 (1996): 9-21.

17. K. Lewin, Field Theory in Social Science (New York: Harper Row, 1951).

18. J. P. Kotter, Leading Change (Boston, Massachusetts: Harvard Business School Press, 1996).

19. Linda Ackerman, "Development, Transition, or Transformation: The Question of Change in Organizations," in Organizational Development Classics, eds. Donald Van Eynde, Judith Hoy, and Dixie Cody Van Eynde (San Francisco: Jossey-Bass, 1997): 45-58.

20. Rosabeth Moss Kanter "The Change Masters," in Organizational Transitions: Managing Complex Change, eds. Richard Beckhard and Rubin T. Harris (Wokingham: Addison-Wesley, 1987). See also D. Nadler and M. Tushman, "Organizational framebending," Academy of Management Executive 3 (1989): 194-202.

21. Levitt and March, "Organizational Learning."