Research Design and Major Issues in Developing Dynamic Theories by Secondary Analysis of Qualitative Data

Abstract

This article discusses the use of the Grounded Theory approach to the analysis of qualitative data to develop dynamic theories. The main objective of the article is to explore the major issues in bringing Grounded Theory and System Dynamics approaches together. The discussion employs an example of secondary analysis of qualitative data as a mixed-method research design for System Dynamics researchers. Findings from this example are organized in a table by research stages. Some of the reported major issues have divided researchers in their fields, such as the theoretical sensitivity issue in the Grounded Theory field and the qualitative and quantitative modeling issue in the System Dynamics field. Other major issues, such as the nature and richness of the dataset, can limit the data analysis and the research outputs. Researchers using the mixed-method should address these issues, and the present article offers solutions for them.

1. Introduction

The many dialogs between qualitative and quantitative researchers over methodological differences have contributed to a broad understanding of the strengths and weaknesses of these methods [1]. These discussions have also led some researchers to combine the strengths of these different methods to improve results [2,3]. Researchers have been integrating quantitative and qualitative methods since the 1980s, calling these integrated approaches mixed methods [4,5]. This paper explores one such effort, employing the mixed-use of Grounded Theory and System Dynamics modeling in a secondary analysis of qualitative data. The main audience of this paper is System Dynamics researchers who want to do a secondary analysis of qualitative data with the mixed method described in this article.

System dynamics is a methodology based on mathematical modeling techniques to explore complex systems. The System Dynamics approach typically results in a mathematical model quantitatively describing the structure and behavior of a complex system. The system dynamics field has combined “the theory, methods, and philosophy needed to analyze the behavior of systems…” since its introduction and qualitative data has been an important part in exploring complex social systems [6]. However, researchers in the System Dynamics field have not taken a significant part in the methodological discussions between qualitative and quantitative researchers. System Dynamics researchers have noted the lack of detailed protocols on mixed methods in the modeling process in the last decade [7] despite the existence of successful publications developed with the mixed methodology [8]. Since then, some studies have begun to address this need [9,10,11,12,13,14,15,16,17,18,19]. This article adds to this emerging line of work by describing new insights into the use of a mixed-method research design and combining Grounded Theory and System Dynamics modeling.

This paper starts with a description of the role of qualitative data in System Dynamics. The importance of the secondary analysis of qualitative data follows this section. The mixed-method research design developed in the previous research [9] is then presented. This mixed-method for doing a secondary analysis of qualitative datasets brings forward some methodological issues.

The main focus of the paper is to describe implementation issues in the use of mixed methods, particularly in the exploitation of existing qualitative data for theory building. In this case this mixed-method approach is applied to a secondary analysis of interview transcripts.

These methodological issues are framed in the iterative research stages of the example research project: research question formulation, data analysis, and theory development. This framework was used to explain the methodological issues experienced in a research project that utilized the role of qualitative data in System Dynamics research.

Qualitative data have been an important information source for the System Dynamics field since its inception. Forrester [6,20] listed information sources in modeling as a mental database, written database, and numerical database. “Database” in this context uses an extended definition. “Data” also use an extended definition: “something that is given from being experientially encountered”, and “material serving as a basis for discussion, inference, or determination of policy” and “detailed information of any kind” [6].

Forrester noted the different sizes of information sources available to researchers. The mental database is the largest of these information source categories. Forrester indicated the importance of the mental database and also acknowledged the lack of its appreciation in the social sciences. The mental database offers important modeling information about the structure, policies, and system behavior [20].

The written database is smaller than the mental database, and some represent a recording of a mental database. Concepts and abstractions are another part of the written database. Given his advanced modeling skills and role as the founder of the System Dynamics field, Forrester’s [20] valuation of different written sources is very important. From this perspective, a secondary analysis of qualitative data, in this case verbatim interview transcripts, is a potentially valuable resource as a source of concepts and abstractions, as well as detailed process descriptions. Forrester values the periodical press more than professional literature, textbooks, or journals. The press is richer in reflecting the current pressures surrounding decisions. Understanding such pressures is important for a system dynamics modeler to understand and describe the behavior of systems correctly.

Abstractions about system structures in the written database are also very important in modeling [20]. Forrester exemplifies the contribution of abstraction in the system dynamics model structure with a mathematical functional form from economics (Cobb–Douglas). As discussed in the coming sections, the timing and nature of the use of abstractions in the mixed method can have important impacts on the conduct and results of a research project.

A significant shortcoming of written records is their filtered nature. Researchers may filter information from their personal perspectives and purposes when converting mental information to a written form. Written records are also “not responsive to probing by the analyst as he or she searches for a fit between structure, policy, and behavior” [20].

A numerical database has the narrowest scope among the information sources since it is missing structure and policies. Numerical data are limited in extracting the cause and effect relationships between variables. On the other hand, a modeler can use numerical data as parameter values, system behaviors, and time-series information for model output comparison.

Forrester [6] values qualitative data residing in people’s minds over the quantitative data. He also notes that it is a “far richer and more informative body of information that exists in the knowledge and experience of those in the active, working world.” He adds that this information source has long been overlooked by the social and management scientists, despite its importance [6].

As valuable as it may be, extracting the modeling information from people is not an easy task. However, doing so successfully is critical since the strength of the System Dynamics approach comes from the fit between “the level-rate-feedback structure” and “the fundamental and universal structure of real social and physical systems” [21]. This fit can be revealed by capturing information from mental models and converting it into System Dynamics models. A series of tools and techniques were developed or adopted to elicit modeling information from mental models [22,23,24], (Richardson and Pugh, 1981; Sterman, 2000; Wolstenholme and Coyle, 1983) such as the Group Model Building approach [25,26].

2. Importance of Secondary Analysis of Qualitative Data

When it comes to analyzing a written database, especially qualitative work with secondary data, capturing modeling information becomes more challenging. Challenges include the limited ability to access the subjects, unclear purposes of the written records, the complexity and richness of the written records, and the substantial effort required to analyze the records.

While interacting with secondary qualitative data, the researcher may not be able to access the subjects to ask follow-up questions due to their unavailability. Lack of conversation with the subjects limits the researcher in understanding their mental models and asking them direct questions for modeling purposes [20]. The purpose of the written records may present another challenge to the researcher. The purpose of the written records may not fit the purpose of the research. While the written records may be rich for the original purpose, they may not be rich enough for modeling. Those who prepared the written records may subjectively filter the content. Overcoming these challenges requires painstaking analysis of the written record and can be very time-consuming.

In spite of these challenges, the availability of secondary qualitative data presents many valuable opportunities for a researcher. Direct access to the mental databases of subjects may not be an option, so secondary analysis of qualitative data may be the only option. For instance, researchers studying the collapse of Maya Civilization did not have an opportunity to organize a group workshop with ancient Mayans. They instead relied on the written records or archeological findings to develop their models [27]. Even for studying more recent events, such as the World Trade Center response and recovery process after the 9/11 attacks in this case, finding research subjects may be hard or costly even though many of them may remain alive. Respondents may also not recall many details years after the event. Below, researchers in this case discuss analyzed transcripts of interviews done with critical actors who participated in the recovery process close to the time of the attacks [9,28]. In some situations, respondents may simply be unavailable. In studying the Federal Open Market Committee’s judgment building process, organizing a group meeting for research purposes was virtually impossible. Thus, the researchers relied on analyzing their publicly available meeting transcripts [29].

3. Need for Tools and Techniques to Analyze Qualitative Secondary Data Analysis

Researchers planning to use qualitative methods for System Dynamics research face a highly varied body of possible analysis methods. In their study on the use of qualitative data for system dynamics, Luna-Reyes and Andersen [7] indicate a “lack of clear description about how and when to use” qualitative data, despite the general agreement about its importance among the modelers. Grounded Theory, along with other qualitative data collection and analysis techniques, such as case study, “interviews, focus groups, Delphi studies, and participant-observer research” can be used in developing System Dynamics models based on qualitative data.

This paper argues for the value of the mixed-use of Grounded Theory and System Dynamics for the secondary analysis of qualitative data as a path through the qualitative analysis thicket. Scholars see potential in this mixed-method [14,30,31], and several successful results have been reported [8,32,33,34]. Researchers have started to provide more clear descriptions on this mixed-method. These descriptions include a codebook and coding rules specific to system dynamics modeling [35], a guide to coding maps from purposive data [29], a framework to connect research designs to coding practices [11], using binary matrices to store causal loop diagrams [36], and an alternative approach for conceptualizing system dynamics models [14]. In the meanwhile, this trend has also brought attention to early approaches, such as The Documentary Coding Method [37].

4. Research Design for Developing Dynamic Theories by Analyzing Qualitative Secondary Datasets

This section describes a research design (Figure 1) for the mixed-use of Systems Dynamic Modeling and Grounded Theory developed in a previous research project [9]. The iterative nature of these approaches prevents an effective graphical presentation of the mixed-method research design and explanations around this presentation. Knowledge of Grounded Theory and System Dynamics approaches are required to explore the research design here to address the potential gaps in this presentation.

This research design consists of three stages: Research Question Formulation, Data Analysis, and Theory Development. There is an iterative relationship among these stages. A research project starts with the Research Question Formulation; an overall understanding of the research goal and heuristic concepts are developed in this stage. Heuristic concepts are the concepts that help “the researcher to focus the attention on certain phenomena in the empirical field” [38]. These concepts range from high to low empirical content. Low-level heuristic concepts have empirically low content, which does not force the analysis down certain paths, as do the hypotheses in deductive quantitative research.

In this stage, a review of the literature and revisiting some generic models or system archetypes is possible, depending on the specific purpose of the research. Although this is the starting stage, it does not end with clear-cut research questions given the nature of the Data Analysis and Theory Development stages. The initial research question may change or new research questions may arise during these later stages. As a result, repeatedly returning to the Research Question Formulation stage, reviewing the literature, or updating the core assumptions and heuristic concepts are considered normal elements of the process.

The core practices of the Grounded Theory mainly shape the Data Analysis stage. The secondary analysis of the qualitative data impacts the sampling process. Theoretical sampling is used to select the relevant text samples from an existing, and usually limited, dataset.

Core coding practices, such as open coding, axial coding, and selective coding, are used to extract the findings, as in any usual Grounded Theory-based research projects. The need to discover a dynamic result is an important distinction in this mixed method. Heuristic concepts and a codebook should include low-level empirical system dynamics concepts, such as the causal relationship, nature of the relationship, stock, flow, and rate of change-over-time. The outputs of the Data Analysis stage can be organized as categories and properties, relations between categories, memos and diagrams, system dynamics model information, and a causal understanding of the phenomenon. While these outputs develop, the researcher revisits the research question, literature, generic models, or system archetypes, and revises the heuristic concepts and codebook, if needed. A schematic representation of how the design and analysis can proceed is shown in Figure 1 below.

A theoretical understanding saturates during the Data Analysis phase, and the researcher typically builds and reinforces this understanding through theoretical concepts. The back-and-forth interaction with the secondary qualitative data continues during this saturation phase, and a final theory iteratively emerges as a result of these interactions.

Sensitivity to the dictation of the ideas and focus on the subjects’ voices from the dataset are the core strengths of the Grounded Theory approach. Having a matured understanding of the subjects and the phenomenon, the researcher can now consider higher-level detailed concepts, such as hypotheses in the use of quantitative research. Testing these concepts may require some additional visits to the previous stages, additional literature reviews, and further data analysis. However, thanks to the researcher’s developed understanding based on the data, these high-level empirical concepts will be less likely to bias the researcher. The researcher will be able to reject or confirm such concepts by approaching them from an in-depth understanding of the subjects’ voices.

This overall summary of the research design does not address many critical issues in the mixed-use of Grounded Theory and System Dynamics approaches. Table 1 organizes these issues and the discussion will continue based on this table.

5. Research Question Formulation

5.1. Goal of Research Question Formulation

Defining the research question is the main goal of the Research Question Formulation stage. The Grounded Theory approach requires a research question formulation that helps the researcher to focus on a particular phenomenon [39]. In this stage, the researcher should avoid assumptions and let them emerge from the data in the later stages.

5.2. Inputs of Research Question Formulation

Potential inputs in this stage are literature (including System Dynamics literature) an existing theory in question, generic models, and system archetypes. However, researchers should carefully adjust the intensity of using these inputs, considering the conflicting goals and concerns of both the Grounded Theory and System Dynamics approaches.

5.3. Outputs of Research Question Formulation

The major outputs of this stage are a research question that will direct researchers toward a phenomenon and heuristic concepts in a starting codebook.

5.4. Issues in Research Question Formulation

There are more issues in this stage than in any other. The sensitive approach to inputs and research problems in the Grounded Theory approach requires consideration of many issues. These issues include how to use the existing literature in the early stages (remaining free of any theoretical preconceptions), theoretical sensitivity, the empirical content of theoretical codes, the falsifiability of statements, and heuristic concepts. These have already discussed extensively in the Grounded Theory field. Other issues emerge from the System Dynamics field, such as using generic structures and system archetypes at this stage.

5.4.1. Using Existing Literature in the Early Stages

The classic Grounded Theory approach minimizes the use of inputs (such as existing literature) in the early stages of research to avoid researcher bias problems [40]. The Grounded Theory approach contradicts the hypothetico-deductive method that “enforces the development of precise and clear cut theories or hypotheses before the data collection takes place” [38] and asks “literally to ignore the literature of theory and fact on the area under study in order to assure that the emergence of categories will not be contaminated …” [41]. This position is one of the most important strengths of the Grounded Theory, allowing researchers to focus on the data under consideration without being biased with the existing theories and literature. This way, data analysis can be done without forcing the data to fit the researcher’s presumptions.

With respect to the idea of being free of any theoretical preconceptions (the section below), some researchers do not fully agree with the treatment of the existing literature on Classic Grounded Theory. Instead, they have developed different versions [42]. Even the original theorists of the Grounded Theory divided their approaches. The Classic (Glassarian) Grounded Theory has a more sensitive approach to the existing literature. On the other hand, Strauss and Corbin [43,44] have a more flexible approach to the use of the literature in the early stages of a research project. Glaser criticizes these trends, including the ones mixing the Grounded Theory with other qualitative data analysis methods, and he describes “the effect of downgrading and eroding the GT goal of conceptual theory.” [45].

This discussion and the distinction between different Grounded Theory approaches are important in the mixed-use of Grounded Theory and System Dynamics approaches. The system Dynamics field is rich with generic models and system archetypes. Knowledge of such generic structures and archetypes may be seen as a bias problem by the Classic Grounded Theory approach. Such a sensitive approach arouses critical questions as to whether a researcher can be free of any theoretical preconceptions.

5.4.2. Being Free of Any Theoretical Preconceptions

Despite their different approaches to the use of the literature in the early stages of a research project, researchers using the Grounded Theory agree on avoiding bias towards some concepts. While supporting the early use of the literature, Strauss and Corbin [44] also emphasize that “a researcher does not begin a project with a preconceived theory in mind unless his or her purpose is to elaborate and extend existing theory.” The question then focuses on whether awareness of literature, a theory, a generic structure or a system archetype poses a risk to develop a preconceived theory in mind and force data into this preconceived idea.

Kelle [38,46] explores the theoretical preconception issue from its roots in positivist epistemology. Francis Bacon and John Locke are the earliest philosophers who adapted the inductive process of theory building while avoiding theoretical preconceptions before analyzing the empirical data. This approach, also called “naïve empiricism” or “naïve inductivism,” was criticized by Immanuel Kant and lost most of its supporters. Kelle discusses that it is impossible to free empirical observation from all theoretical influence [38]. Since it is not possible to build theory without accumulated knowledge, neglecting existing theories demands an empty, rather than an open, mind. Researchers typically bring their theoretical lenses to their research. Dropping these lenses negates their ability to perceive, observe, and describe meaningful events [38].

In their article on improving qualitative methods, Brower, Abolafia, and Carr [47] also recommend researchers to link theory with the data early and often. They argue that researchers must be informed of theories and continue to review the literature even in the data analysis stage. This awareness does not mean giving up theoretical sensitivity, since “researchers who do theoretically sensitive coding of social, political, and economic conditions in field notes and interview transcripts will reveal more theoretically powerful pictures of causality.” [47]. Using the literature, theoretical concepts, generic models, and system archetypes remain important challenges while being theoretically sensitive in a data analysis. Determining the useful codes at the early stages of a research project can address such a challenge while avoiding the bias trap.

5.4.3. Theoretical Sensitivity and Generic Dynamic Structures

Exploring the “theoretical sensitivity” and “theoretical concepts” discussions in the Grounded Theory field can help determine the use of generic dynamic structures during a research project. In the original introduction of the Grounded Theory, the “theoretical sensitivity” concept of Glaser and Strauss [41] accepts the use of theoretical concepts in advance by addressing the researcher’s ability “to reflect upon empirical data with the help of theoretical terms.” This view has evolved into two different understandings of “theoretical sensitivity”. Glaser [48] posits the “theoretical codes” concept, which accepts the theoretical concepts at the researcher’s disposal independent from data collection and data analysis. However, “theoretical codes” should “earn their way into the theory as much as substantive codes”, so they should not be preconceived but should instead emerge from the data [40]. In the other version, Strauss and Corbin take a more open position by noting that “all kinds of literature can be used before a research study is begun…” [39].

While discussions on the use of theoretical concepts created different versions of the Grounded Theory approach, the System Dynamics field took a very open position towards the use of concepts like “generic structures” and “archetypes”. Generic structures played an important role in storing and generalizing insights in the System Dynamics field [49]. Archetypes have been used to “generate understandings in new application domains and systems” [50]. According to Lane and Smart [49], “Forrester had always advocated building a general model or theory first, and then modifying it to fit the particular situation under study as the preferred method for building any system dynamics model.”

In the early stages of a research project, developing theoretical codes from the “generic structures” of the System Dynamics field should be done while respecting the “theoretical sensitivity” idea of the Grounded Theory field. Such respect can be developed by evaluating these theoretical codes from the “empirical content” and “falsifiability of statements” perspectives.

5.4.4. Empirical Content of Theoretical Codes

The roles of theory and hypotheses in the Grounded Theory methods are distinct from those of typical quantitative research methods. These distinctions are central to understanding the empirical content of qualitative statements as data. Quantitative research defines a hypothesis as a testable claim about a relationship between an independent variable and dependent variable(s). The focus in such research is on falsifying an empirically contentful statement [51]. On the other side, a hypothesis serves a different purpose in qualitative research. Especially in the Grounded Theory approach, hypotheses emerge during the interaction with the data, and initial hypotheses are vague ideas about relations that are contrary to the hypotheses in the quantitative studies.

Theoretical preconceptions typically impact the ways that researchers (including the qualitative ones) structure empirical material. While these preconceptions can be noted in clear-cut falsifiable statements in quantitative research, they are not represented in the same way in qualitative research.

Kelle [46] indicates this contrast and notes the confusion of using the term “hypothesis” in both qualitative and quantitative fields. He suggests that “instead of calling [these vague ideas] hypotheses, one should rather call them hypotheses about what kind of propositions, descriptions or explanations will be useful in further analysis.” He [46] proposes that “they should be referred to as ‘heuristic concepts’ which can be used to formulate ‘orientation hypotheses’ [52]”.

A quantitative researcher’s theoretical knowledge forms “a fully coherent network of explicit propositions from which precisely formulated and empirically testable statements can be deduced” [38]. On the other hand, having hypothetical inferences enable a creative process to combine new empirical facts with existing theoretical knowledge for a qualitative researcher. Kelle [38] notes these issues and indicate that a qualitative inquiry “should constitute (a sometimes only loosely connected) ‘heuristic framework’ of concepts (or ‘coding families’), which helps the researcher to focus the attention on certain phenomena in the empirical field.”

5.4.5. Falsifiability of Statements

In the development of a heuristic framework, quantitative research’s falsifiability of statements approach can help qualitative researchers to identify the problematic theoretical concepts or codes that can force data into preconceived ideas. The hypothetico-deductive framework dictates how a quantitative researcher will precisely formulate falsifiable concepts or hypotheses. Otherwise, such concepts would be problematic in measuring a relationship quantitatively. However, this problematic nature is valuable in the Grounded Theory approach, because “theoretical concepts with low empirical content … can play an extremely useful role … since they do not force data into a Procrustean bed” [38]. If a statement or a theoretical concept is not falsifiable, then such a concept can provide a flexible idea about the problem and can be used to explore various kinds of empirical phenomena during the analysis.

5.4.6. Heuristic Concepts

Concepts can be categorized based on their empirical content from low to high by examining their falsifiability. These categories are theoretical notions (low empirical content), common sense categories (low empirical content), and falsifiable concepts (high empirical content) [38,46,53].

The empirical content of theoretical notions is so low and broad that these concepts seriously limit the deduction of falsifiable propositions from them. They can be derived from “grand theories” and are “highly abstract concepts about the relations between actors or between actors and society in general” [38,46]. Blumer [54] calls them “sensitizing concepts” that “lack precise reference and have no benchmarks which allow a clean cut identification of a specific instance” as opposed to “definitive concepts” that “provide prescriptions of what to see”. Applying codes generated from a specific theory can be easier in data analysis, but this also brings the risk of missing important theoretical concepts from other theories. That’s why using different or even competing theoretical perspectives in developing the codes can result in better theoretical coverage [38].

Common sense categories are also heuristic concepts with low empirical content, which can guide the researcher in the analysis. These categories can be identified by relying on “theories of the members of the investigated culture” [46] and drawing on general knowledge [53].

Falsifiable concepts have high empirical content and are, at least in principle, falsifiable. These concepts are the desired concepts in developing a hypothesis in the Hypothetico-Deductive approach. On the other hand, their use in the early phases of data analysis in the inductive approach has a very high risk of forcing data into preconceived ideas. However, these concepts can be developed in the advanced stages of the analysis.

5.5. Solutions for Research Question Formulation

These serious issues can be addressed by being sensitive to theoretical biases, avoiding concepts with high-level empirical content at the beginning of the research, and developing heuristic concepts to better explore the phenomenon in question.

5.5.1. Be Sensitive to the Theoretical Bias

The researcher should appreciate the sensitivity required to deal with theoretical bias in the mixed-use of Grounded Theory and System Dynamics. This sensitivity will help the researcher to focus on listening to the voice of the subject, while the theories, especially the new patterns and relations, will emerge based on this voice during the analysis.

The Classic Grounded Theory sees early literature review as a threat to the fundamentals of Grounded Theory [45] for the risk of literature biasing the data analysis. Such risk already exists for an experienced researcher well knowledgeable on the existing theories in a given research domain. A novice researcher does not carry such a risk thanks to such a researcher’s limited knowledge. Avoiding the extensive literature review will also prevent the risk to develop an understanding based on the literature, but not the analyzed data. On the other hand, a novice researcher faces the risk of getting lost in the analysis due to a lack of guiding concepts. Even with the theoretical codes [48], a novice researcher may have trouble in effectively using the coding families. A predefined coding paradigm can address this problem [53].

5.5.2. Avoid Concepts with High-Level Empirical Content

Using theoretically sensitive coding families can help the researcher derive categories grounded in data. However, distinguishing the difference between theoretically useful and forceful codes may be a challenge for novice researchers. In order to address this challenge, the researcher should use the methodological concepts such as ‘falsifiability’ to distinguish the level of ‘empirical content’ of concepts or statements [53]. The coverage of a concept can be another focus in choosing the right heuristic concept. If a concept excludes interesting processes or events from being analyzed, it could be problematic as a heuristic concept.

5.5.3. Develop Heuristic Concepts

Regardless of the researcher’s experience, research should focus on a specific research question to avoid wasting time in data collection and analysis stages [47]. Identifying such questions typically requires an initial understanding of the phenomenon. This understanding naturally guides the researcher in the analysis process. A heuristic framework can help the researcher in properly analyzing the data. These heuristic concepts should be generic enough to not bias the research, but specific enough to help the researcher to explore the problem. Theoretical notions and common-sense categories guide the researcher in coding. These concepts with low-level empirical content can come from different theoretical traditions, such as the System Dynamics field [53]. Core System Dynamics concepts such as the stocks, flows, loops, causal relations, or nature of these relations are concepts with low-level empirical content that cannot force the researcher to work with a certain preconception. On the other hand, they are required components to construct a system dynamics model to explain the phenomenon in question. Similarly, many archetypes can be considered empirically low-level concepts. The researcher should examine these concepts before adding them to a heuristic framework by checking their falsifiability and inclusiveness.

6. Data Analysis

6.1. Goal of Data Analysis

The mixed-use of Grounded Theory and System Dynamics impacts the goal of the data analysis stage. As in all research methods, the data analysis stage is designed to analyze the data to get answers to research questions. From the Grounded Theory perspective, the goal is to understand the problem in a question while letting the research answers emerge from the data. The System Dynamics perspective shapes this goal to determine the System Dynamics oriented artifacts typically required to build a dynamic model.

6.2. Inputs of Data Analysis

6.2.1. Research Question

The outputs of the Research Question Formulation stage serve as inputs for this stage. The research question developed in the previous stage helps the researcher to focus on a particular phenomenon. New research questions can emerge during the data analysis, or existing research questions can change. The data analysis process may reveal some interesting questions and findings, even in the cases with a mismatch between the original research question and the content of the secondary qualitative data. The researcher can modify the focus of the research based on these new findings.

6.2.2. Heuristic Concepts

The heuristic Concepts developed in the previous stage play an important role in labeling and categorizing the emerging concepts and the relationships between these concepts. Heuristic concepts come from both the domain under study and the System Dynamics core concepts.

System Dynamics related coding categories can be depicted based on the Theoretical Notions and Common-Sense Categories. For example, Wrightson [37] points to three core code categories in the most basic structure: “Cause Concept,” “Linkage,” and “Effect Concept.” Glaser’s [48] “Six Cs” coding family (Causes, Context, Contingencies, Consequences, Covariances, and Conditions) covers these code categories. However, the goal of a System Dynamics researcher is to use these concepts to build modeling artifacts, such as cognitive maps, causal loop diagrams, and simulation models. Thus, these concepts are organized in order to reach this goal. For example, linkage types are defined as coding categories. Positive and negative linkage types are the most common (and almost default) linkage types in the System Dynamics field. Positive linkage refers to changes in the same direction in the values of the cause and effect concepts, while negative linkage refers to changes in the opposite direction. These linkage types are used in mixed methods as core codes [35]. Alexrod [8] uses an extended list of linkage types (a total of seven types) to explain the relationships in his cognitive maps.

Heuristic concepts can be extended to include other concepts, thereby explaining probability relationships [37], degree [48], feedback structures, fundamental modes of dynamic behaviors [24], stock and flow concepts, and other theoretical notions and common-sense concepts. While an experienced System Dynamics researcher may not need to list all the core concepts, such a list may be very useful for a novice researcher.

6.2.3. Developing Theoretical Concepts and Related Information Needs

Data Analysis and Theory Development stages are overlapping in discovering the findings and developing a theory around these findings. Given the iterative nature of these processes, emerging theoretical concepts from the Theory Development stage have become inputs to the Data Analysis stage. The researcher typically seeks additional information, such as the specific properties of a theoretical concept. In the System Dynamics context, this information typically needs to come from the modeling parts.

6.3. Outputs of Data Analysis

The data analysis stage results in outputs similar to the outputs of a Grounded Theory research project. The most important difference is the organization of outputs from the System Dynamics perspective. Typical outputs are coded categories, as well as their properties, relations between categories, memos and diagrams, System Dynamics Modelling information, and causal understanding of the phenomenon.

In the mixed-use of Grounded Theory and System Dynamics, the researcher attempts to develop a causal understanding of the phenomenon by finding information on the necessary parts of the System Dynamics models.

6.4. Issues in Data Analysis

Issues in the Data Analysis stage focus on the dataset and the data analysis process in terms of their impact on extracting necessary system dynamics modeling information from a secondary qualitative dataset. The nature of the dataset, the richness of the dataset, and the length of the data analysis are the major issues explored here. All these issues are common to both the Grounded Theory and System Dynamics fields.

6.4.1. Nature of the Dataset

Various types of documents, such as diaries, newspapers, interview transcripts, or meeting transcripts, can be used in data analysis as secondary data. Analysis of a qualitative secondary dataset severely restricts the researcher to written information in understanding the subjects. The researcher typically does not have a chance to interact with subjects in-person to ask questions to better understand their stance on an issue. Sometimes, documents may be written through the author’s subjective lenses. More importantly, these documents may not be originally prepared for the phenomenon of interest to the researcher.

6.4.2. Richness of the Dataset

Another challenge is the richness of the dataset for research. The dataset should be rich enough for the researcher to explore the phenomenon in question and collect necessary modeling information. This exploration can be especially challenging in the secondary analysis of qualitative data with large or diverse datasets commonly prepared for purposes other than the research question in mind.

6.4.3. Length of the Data Analysis

Analyzing each word, sentence, and paragraph is a time-consuming activity in qualitative data analysis. Depending on the complexity of writing and the length of a document, even relatively short passages may take many hours or days. In the mixed-used of methods, the focus is not only on the emerging findings to understand the problem but also collecting modeling specific information to describe it from the Systems Dynamics perspective. This goal requires extra effort and attention during the data analysis, which slows the research further.

6.5. Solutions in Data Analysis

6.5.1. Evaluate the Dataset Broadly

Broadly evaluating the secondary data helps to deal with the nature and richness of dataset issues. Broad evaluation is about understanding the content of the dataset to see if it can be used to answer the research question in the researcher’s mind. Using flexible research questions that are open to change allows one to steer through limitations of the secondary data [55].

6.5.2. Focus on Understanding the Problem (System) in Question

The challenges associated with a secondary dataset along with the taxing data analysis process can distract a researcher. The dataset used in the research may not be perfect, and there may be too many heuristic concepts, even at the beginning of the project. However, the researcher should not forget the main goal of exploring a problem in question. Research should focus on understanding the problem even in the absence of some critical modeling information. This problem is discussed further in the Theory Development section.

6.5.3. Use a Core Codebook

A core codebook developed based on Theoretical Notions, Common-Sense Concepts, and System Dynamics Modeling Concepts helps the researcher to manage the coding process better. As the coding process starts to reveal interesting findings, the researcher relies on the core codebook to check and find the critical information needed for the modeling.

7. Theory Development

7.1. Goal of Theory Development

The goal of the theory development stage is to develop a dynamic theory to explain the components and behavior of the system.

7.2. Inputs of Theory Development

The Data Analysis and Theory Development stages overlap in discovering findings and developing a theory around these findings. Given this iterative nature, while outputs from the Data Analysis stage become inputs for the Theory Development stage, some of the outputs of the Theory Development stage become inputs to the Data Analysis stage.

The researcher iteratively reviews the outputs, such as categories and properties, relationships between categories, memos and diagrams, system dynamics modeling information and causal understanding of the phenomenon. When a gap is identified, the researcher goes back to the data and searches for a finding to close that gap.

7.3. Outputs of Theory Development

A researcher can generate one or more of the listed outputs of the Theory Development stage. The number of outputs depends on the success of the data analysis stage, given the nature and richness of the data and other factors.

7.3.1. Substantive Theories to Explain Specific Cases

A researcher using the Grounded Theory approach develops a theory with “a set of well-developed categories (e.g., themes, concepts) that are systematically interrelated through statements of relationship to form a theoretical framework that explains some relevant social, psychological, educational, nursing, or other phenomenon [44].” The mixed-use of this approach with the System Dynamics approach should at least aim to develop a theory at this level.

7.3.2. Cognitive Maps

Incorporating the core concepts from the systems and related fields, a researcher can develop cognitive maps to explain the decision making logic of an individual or a group [8] as a result of the mixed method use. Alexrod [8] uses basic elements in his cognitive maps: points (concepts a person uses), arrows (causal links between these concepts), and plus/minus notations (nature, positive or negative, of the causal links). The secondary analysis of a qualitative dataset can acquire these basic elements.

7.3.3. Causal Loop Diagrams

A causal loop diagram has elements similar to cognitive maps. It shows variables (rather than focusing on concepts or ideas), causal relationships between variables, and the nature of these relations. Causal loop diagrams are developed to represent the complex and dynamic nature of systems and their behaviors. Complex systems typically have closed sequences of causes and effects that a causal loop diagram can effectively visualize. The mixed-method approach can result in causal loop diagrams representing the system or problem’s complexity [56].

7.3.4. Reference Modes

A reference mode is a time series graphical model describing the historical behavior and future trend of a variable [57]. These graphs offer important findings of the behavior of a given variable and related system in an organized and intuitive way. The mixed-method use can generate reference modes as outputs to describe a phenomenon.

Reference modes have an important place in the System Dynamics field for understanding the systems. Researchers in this field frequently explain system behaviors with these graphs. Although dynamic behaviors have been discussed in other fields, the use of reference modes is typically not common in explaining these dynamic behaviors. When used, reference modes can effectively organize one’s findings and clearly show the patterns for both past and future behavior. The project involvement of different actors over time [58] and assimilation gap in a software process [59] are good examples of reference modes developed outside the systems school.

Developing reference modes as a result of the mixed method use can make an important contribution to the understanding of a phenomenon. The reference mode concept and its properties can also become part of these heuristic concepts. This concept can guide a researcher in searching for the dynamic behaviors of variables in a dataset.

7.3.5. Fully Developed System Dynamics Models with Dynamic Hypotheses

A fully developed system dynamics model is a simulation model. It includes critical variables and relationships between variables in a mathematical form. This system shows dynamic hypotheses based on these definitions. Retrieving all the modeling information, including mathematical formulas of the causal relationships, is a challenge in the secondary analysis of a qualitative dataset. However, an experienced researcher can recognize the common dynamic structures in a phenomenon and use system archetypes as a guide to build a model to reflect this understanding. Dynamic behaviors generated from a model can be compared to the dynamic behaviors observed in the dataset.

7.3.6. Extended and Elaborated Generic Structures

Extending and elaborating a generic structure may be the goal of a research project. The framework of a generic structure can be used to analyze a specific case discussed in a qualitative dataset. Heuristic concepts generated from the generic structure can be used to explore a phenomenon. As a result, these generic structures can be extended or elaborated.

7.4. Issues in Theory Development

The type of research output and the use of generic models and archetypes are issues that mainly emerge from the System Dynamics field.

7.4.1. Type of Research Output

As discussed in the previous section, the mixed method use can generate different outputs. Cognitive maps, causal loop diagrams, and some reference modes are qualitative system dynamics models. Simulation models are quantitative system dynamics models. The value of these outputs has been discussed in the system dynamics field. While one school of thought values qualitative outputs [23,60], another school of thought values the quantitative output as the ultimate output of a research project [61,62,63,64].

The disagreement between schools is a critical issue, since the lack of information is one of the core problems in developing a quantitative model. Given the limitations of a secondary qualitative dataset, extracting all modeling information is challenging. A full extraction is very unlikely, and the researcher may end up with “a small number of concepts (model variables) and relationships among them” [14]. The quantitative school indicates that a simulation model can still generate important insights even under uncertainties. Missing information can be found from other sources or can simply be hypothesized to fill the gaps in the modeling information. Simulation models offer an important benefit for testing an emergent understanding.

Several systems researchers successfully demonstrated that they could generate quantitative models by analyzing qualitative data with the Grounded Theory approach [32,34,65]. These dynamic theories grounded on qualitative data demonstrate the ultimate capability of the mixed-use of the Grounded Theory and System Dynamics.

7.4.2. Use of Generic Models/Archetypes

Generic models and archetypes from the System Dynamics field can play an important role in developing theories. As findings emerge from the data, tentative theories may develop resembling certain generic System Dynamics behaviors and structures. Such similarity may prompt a researcher to prematurely explain the phenomenon through the lenses of these generic models. However, the researcher should avoid forcing these findings into generic structures at this stage.

7.5. Solutions in Theory Development

7.5.1. Every Finding Is Important

The mixed-method use can lead to different outputs, and each output is valuable in describing a phenomenon. The secondary analysis of qualitative datasets is an already challenging process with many limitations. Seeking modeling information from the Systems Dynamics perspective makes it more challenging. A researcher may not successfully develop a working simulation model due to the lack of information. Analyzing a large amount of data may lead to the discovery of a few concepts and relationships that can lead to the development of simulation models [14]. As discussed in the Importance of Secondary Analysis of Qualitative Data section, a secondary qualitative dataset provides an opportunity to attain a research setting that may not be available otherwise. Failing to develop a simulation model should not be seen as a failure since the mixed method use can still provide a valuable understanding of the phenomenon through other outputs. Cognitive maps, causal loop diagrams, and reference modes can describe the important findings of a phenomenon. If a researcher is motivated to develop a simulation model, these qualitative outputs also play an important role in the modeling process.

7.5.2. Ground Your Dynamic Theory on the Data

Regardless of the type of output generated, it should be grounded in data as a result of the inductive data analysis process. Parts of the outputs should be developed based on the codes that emerged from data. The grounding process prevents the researcher from forcing the data into a generic model. This is an important contribution of the Grounded Theory approach in analyzing secondary qualitative datasets.

7.5.3. Report Gaps and Hypothesized Fillers

Given the limitations of the secondary analysis of qualitative data, retrieving all the modeling information may not be possible in many cases. Following the quantitative school of thought in the System Dynamics field, a researcher may decide to develop a simulation model even with some gaps in the modeling information. To fill these gaps, several strategies, such as using generic behaviors and structures, using similar cases discussed in literature, or simply hypothesizing ideas, can be employed.

A working simulation model can provide additional insights into the problem. The findings can be tested with the hypothesized information, thereby filling the gaps. These gaps and the reasoning behind related the modeling decisions should be explained in detail, and different alternatives should be provided to show the limitations of the model.

8. The Mixed Method Use Example: World Trade Center Research

The mixed-method was implemented in a research project, referred to here as “WTC research,” to learn critical lessons from a very valuable qualitative dataset. This dataset is taken from an empirical study exploring the role of information technology used by government agencies and other organizations during the response to and recovery from the attack on the World Trade Center (WTC) on 11 September 2001 [66]. Researchers at the Center for Technology in the Government (CTG) at the University of Albany-SUNY reached out to 29 responders and interviewed them over ten months in 2002–2003. These responders were critical decision-makers during the aforementioned events. They provided rich narratives of interagency collaboration in the context of using information, technology, and interagency coordination in response to the attacks. Compiling such a dataset was difficult to accomplish even a few years after the event. It would be almost impossible today, because these critical decision-makers have moved to different organizations/positions, retired, or possibly passed away.

This section discusses the WTC research based on the framework explained in this article. It is not designed to report the findings from the WTC research. This section has been developed to showcase the mixed method issues experienced in a particular research project.

8.1. Research Question Formulation

8.1.1. Goal of Research Question Formulation

Collaboration among researchers led to the development of the research question in this research project. Researchers who were familiar with both the WTC dataset and the generic dynamic theory agreed on the potential value of research bringing these parts together. The WTC dataset consists of rich narratives on interagency collaboration in an extreme event. The generic dynamic theory was originally developed in a research project on another interagency collaboration under normal circumstances. Exploration of the WTC events from the generic theory lenses could reveal important insights about interagency collaborations. As a result of these discussions, the research goal became to explore the technical processes and social accumulations in the World Trade Center response and recovery process from the perspective of a generic dynamic theory on socio-technical processes.

8.1.2. Inputs of Research Question Formulation

The generic dynamic theory on socio-technical processes [67] was the most important input to the Research Question Formulation stage. This theory was developed from a research project on an interagency information integration project among New York State agencies. The core of this theory was visualized in a stock-flow diagram (Figure 2). This diagram describes the generic processes that create artifacts in a social process. The model in Figure 2 is the updated version of the original model with formatting changes, such as added polarities.

8.1.3. Outputs of Research Question Formulation

The research question was developed at this stage by discussing and exploring the potential application of the generic dynamic theory on socio-technical processes to the WTC dataset. Several interviews were analyzed to ensure that the dataset would be rich enough to come up with a substantive theory to explain the role of social accumulations in the technical processes of the WTC response and recovery process.

Using the core concepts of the generic theory on the social-technical processes depicted in Figure 2, the researchers developed the following heuristic concepts:

• Process, social process, practice, social practice

• Social accumulation

• Social group efficiency

• Social group effort

• Feedback loop, causal relationship

• Artifacts

8.1.4. Issues in Research Question Formulation

The literature review focused on the literature related to the generic dynamic theory, such as literature on socio-technical processes.

Theoretical notions and common-sense categories were retrieved from the generic theory. These concepts, such as process, social accumulation, social group effort, and artifacts, were highly abstract sensitizing concepts that were difficult to bias the researchers towards a preconceived idea, but effective enough to keep their focus on the research problem.

8.1.5. Solutions for Research Question Formulation

The generic dynamic theory has several dynamic hypotheses on socio-technical processes. The dynamic hypothesis concept is a system dynamics concept that explains a structure that generates dynamic behavior over time (reference modes) [68]. The generic dynamic theory has the following dynamic hypotheses:

• P1:. Social practice causes social accumulation.

• P2:. A reinforcing feedback loop exists between social practice and social accumulation through individual/group effectiveness that builds social effectiveness.

• P3:. A reinforcing feedback loop exists between social practice and social accumulation through individual/group effort that grows motivation.

• P4:. A balancing feedback loop exists in that, as social practice changes Artifact 1, these changes affect social practice back through individual/group effectiveness.

• P5:. A balancing feedback loop exists in that, as social practice changes Artifact 1, these changes affect social practice back through individual/group effort.

• P6:. A balancing feedback loop exists in that, as social practice changes Artifact 2, these changes affect social practice back through individual/group effort.

• P7:. A feedback loop exists in that, as social practice changes Artifact 2, these changes affect social practice back through individual/group effectiveness.

• P8:. Independently accumulated social accumulations affect social practices through individual/group effectiveness.

• P9:. Independently accumulated social accumulations affect social practices through individual/group effort.

• P10:. Social accumulations have initial values that accumulate independently of subjected social practice.

These dynamic hypotheses have more empirical content than the heuristic concepts referred to earlier. For this reason, the researchers initially avoided coding these hypotheses and included them instead in the later phases of the data analysis and theory development.

8.2. Data Analysis

8.2.1. Goal of the Data Analysis

In the data analysis stage, the researchers used the Grounded Theory processes to analyze the WTC data from a generic dynamic theory perspective. Their goal was to utilize the core coding processes to analyze the qualitative data and develop an understanding of the role of social accumulations in the technical processes.

8.2.2. Inputs of Data Analysis

Generic research question and heuristic concepts identified from the generic dynamic theory were initial inputs in the data analysis stage.

Initially, the focus was on the identified heuristic concepts. Social accumulations, artifacts, processes, and the causal relationships between them started to emerge from the data analysis. The researchers also tried to extract modeling information by modeling objects, such as reference modes and causal relationships.

8.2.3. Outputs of Data Analysis

Classical outputs of the Grounded Theory approach, such as categories and properties, relationships between categories, and memos and diagrams, were also generated in this research project. Since the focus was on the causal understanding of the WTC collaboration phenomenon through a generic dynamic theory lenses, the technical processes, related social accumulations, and causal relationships between them were identified. This understanding was reflected in the generic dynamic theory by updating the graph summarizing the theory.

8.2.4. Issues in Data Analysis

The WTC dataset was the main source of issue in the data analysis stage. The interviews were done by multiple researchers with open-ended questions focusing on information technology development and use during the emergency. Although researchers did not specifically ask the technical processes and associated social accumulations, many participants provided rich descriptions of their experience, given the extraordinary nature of the events. On the other hand, responders participated in different projects and their experiences were not focused on a single technical process.

Theoretical sampling was also limited to the available interview transcripts. Data analysis started with the interviews recommended by the original WTC researchers. The researchers analyzed all the interviews at the end. While 29 lengthy interviews brought much material to be analyzed, the researchers did not have a chance to do additional interviews to ask further questions.

8.2.5. Solutions for Data Analysis

Discussing the fit between the WTC dataset and the research goals with the original WTC researchers guided researchers to better evaluate the usefulness of the dataset. Researchers tested the dataset before putting serious effort into the research project by broadly evaluating the content and initially analyzing several interviews recommended by the original researchers.

Having a generic dynamic theory helped the researchers to better focus on the core research problems and concepts. These heuristic concepts were part of the core codebook. The codebook grew as the new concepts emerged from the data.

During the analysis, it became clear that the data analysis process would not generate enough information to build a simulation model, given the lack of focus on a specific process or a social accumulation. However, focusing on understanding the role of social accumulations in the technical processes helped the researchers describe the phenomenon in question and also demonstrate their findings by using mainly qualitative models.

8.3. Theory Development

8.3.1. Goal of Theory Development

The goal in the Theory Development stage was to develop a theory explaining the role of social accumulations in the technical processes during the WTC response and recovery process. In this way, the generic dynamic theory on socio-technical processes could be extended to the WTC case.

8.3.2. Inputs of Theory Development

As the new technical processes, artifacts, or social accumulations emerged from the data analysis, they became part of the theory development process. The outputs coming from the data analysis stage were used to revisit the developing theoretical concepts iteratively.

8.3.3. Outputs of Theory Development

By using the mixed-method approach, the researchers were able to develop a substantive theory of socio-technical processes during the WTC response and recovery process. These findings focused on three major technical processes, and different social accumulations emerged for each technical process.

Stock-and-flow diagrams similar to the generic dynamic theory were used to summarize findings for each major technical process (Figure 3).

Some of the findings were rich enough to describe dynamic behavior over time for a variable. Reference modes for such variables were used to report the findings (Figure 4). Unfortunately, the analysis did not generate such graphs for most of the variables in the models.

8.3.4. Issues in Theory Development

The outputs generated with the mixed method were mainly limited by the nature and richness of the data. The researchers were able to come up with a substantive theory on the role of social accumulations in technical processes. They created multiple causal loop and stock-flow diagrams to explain some of their findings. They were able to extract several dynamic behaviors and reflected them in reference mode diagrams. Overall, the generic theory was extended to the WTC response and recovery process. Findings were organized around the generic dynamic theory.

8.3.5. Solutions for Theory Development

Researchers valued every finding from the analysis and did their best to reflect them in the System Dynamics oriented outputs. The research project resulted in many new research questions and information needs. Having in-person contact with the participants would answer some of these questions. Knowing the value of the dataset for such an extraordinary event, the researchers still focused on the gains generated by this long analysis process.

8.4. Implications of the Mixed Method in This Research

Using the mixed method in this research yielded some advantages and disadvantages at each research stage. When formulating the research question, the mixed method required attention be paid toward issues from both fields. A generic dynamic theory from the System Dynamics field was an important framework to be extended to an emergency situation. This framework introduced some core theoretical concepts and literature to the research formulation process. The Grounded Theory approach forced the researchers to evaluate the empirical content of these theoretical concepts at the beginning of the project to avoid forcing the data into a preconceived idea.

Using the mixed method in the data analysis stage required extra efforts to seek the System Dynamics modeling information for the emerging variables and the relationships between them, including the generic dynamic theory to develop some System Dynamics oriented research outputs, such as causal diagrams, reference modes, and a working simulation model increased the length of the data analysis effort. Researchers also experienced the limitations of the qualitative dataset in terms of extracting enough modeling information for a simulation model. The Grounded Theory process needs a significant amount of data [14] and effort [12] to generate enough information for modeling, as reported in the literature.

In the theory development stage, the modeling information that emerged from the data analysis process was not enough to develop a simulatable model. The Grounded Theory approach by itself could yield a substantive theory explaining the role of social accumulations in the socio-technical processes during the WTC response recovery process. However, using the mixed method increased the output expectations by describing such a substantive theory with the System Dynamics research outputs.

Having a generic dynamic theory framework helped the researchers to organize the findings around it. The researchers extended the generic dynamic theory framework for the WTC case by identifying the variables in three core socio-technical processes. Figure 3 illustrates one of these processes, IT development. However, the behavior of a few variables in these processes was described with reference modes due to the lack of information in the dataset. The System Dynamics lenses helped to describe some parts of the substantive theory in the reference modes, as in Figure 4.

9. Conclusions

The System Dynamics field has had a rich history in using qualitative data to study complex social systems since its introduction. The need for clear-cut data analysis protocols for qualitative datasets has been acknowledged [7] in this field. This paper analyzed this need by exploring the methodological issues in the mixed-use of Grounded Theory and System Dynamics in the secondary analysis of qualitative datasets.

The System Dynamics field has already had some protocols developed to retrieve modeling information from mental databases of subjects. However, such protocols mainly rely on interaction with these subjects. On the other hand, qualitative datasets such as interviews or meeting transcripts can offer very valuable insights into critical events. We had such a case when we received interview transcripts on the role of information technology during the WTC response recovery process. We had to do a secondary analysis of this qualitative dataset.

Researchers in various fields are using the secondary analysis of qualitative datasets to understand phenomena. The System Dynamics field has the additional goal of explaining the phenomena in question with System Dynamics outputs. This additional goal requires the retrieval of modeling information from the dataset, which produces additional methodological challenges. We faced the same challenge when we applied the generic dynamic theory lenses to the WTC dataset.

Researchers have been interested in analyzing such qualitative datasets to develop System Dynamics oriented outputs. However, several critical issues await these researchers in every stage of the mixed method use. Three main stages are identified here: Research Question Formulation, Data Analysis, and Theory Development.

In the Researcher Question Formulation stage, researchers have to appreciate the sensitivity of the Grounded Theory approach towards existing literature and preconceived ideas carried into the analysis. Such concerns can be addressed by acknowledging this bias, excluding concepts with high empirical content from initial data analysis, and developing heuristic concepts that do not bias but guide the researcher.

In the Data Analysis stage, the nature and richness of the qualitative dataset can limit the data analysis. The Grounded Theory analysis requires a serious effort to analyze qualitative datasets. Researchers focused on developing System Dynamics oriented outputs may find such datasets not capable enough to generate dynamic insights. However, these issues can be addressed by evaluating the dataset broadly at the beginning for its fitness to the research goals.

In the Theory Development stage, research should value all findings and outputs. A simulation model may offer significant value in explaining a phenomenon. However, other outputs, such as causal loop diagrams or reference modes, can also be used to describe findings. These outputs can also guide the researcher to develop a simulation model, which may not be fully grounded on the findings. In this case, the researcher should indicate the gaps and how they were filled.

The mixed-method brought advantages and disadvantages to the WTC research project. For example, the Grounded Theory approach brought an extra sensitivity to the use of theoretical concepts at the research question formulation stage. The System Dynamics approach required the extraction of modeling information in the data analysis stage. Since the interviews were not originally done with the modeling information or the generic dynamic theory in mind, extracting such information required an extra effort. Given the limited richness and length of the dataset for modeling purposes, the data analysis process did not yield enough modeling information. Theory development then focused on developing a substantive theory through the lenses of the generic dynamic theory and explaining some particular findings with the System Dynamics research outputs, such as reference modes and causal loop diagrams.

Author Contributions

Conceptualization, B.K.A., A.M.C., and S.G.; Methodology, B.K.A., A.M.C., and S.G.; Validation, B.K.A., A.M.C., and S.G.; Formal Analysis, B.K.A. and A.M.C.; Investigation, B.K.A. and A.M.C.; writing—Original Draft Preparation, B.K.A., A.M.C., and S.G.; writing—Review & editing, B.K.A., A.M.C., and S.G.; Visualization, B.K.A.; Project Administration, B.K.A. and A.M.C.

Funding

This research received no external funding.

Acknowledgments

The authors would like to thank David F. Andersen and Deborah L. Andersen from the University at Albany for their valuable contribution during the research process.

Conflicts of Interest

The authors declare no conflict of interest.

Figures and Table

Figure 1. Research design [9].

Figure 2. Updated generic processes creating technical artifacts in a social process [67].

Figure 3. A model summarizing the results for a socio-technical process [28].

Figure 4. A reference mode describing the change in social accumulation [28].

Table 1

Research stages and critical parts.

Research Stages/Parts	Research Question Formulation	Data Analysis	Theory Development
Goal	Define a research question	Analyze Data to Understand the System in Question and Extract Modeling Information	Develop a dynamic theory to explain the components and behavior of the system
Inputs	Literature/SD Literature Generic Models/System Archetypes An Existing Theory to be Tested	A General Research Question Heuristic Concepts Developing Theoretical Concepts and Related Information Needs	Data Analysis Outputs
Outputs	Research Question Heuristic Concepts	Categories and Properties Relations Between Categories Memos and Diagrams System Dynamics Model Information Causal Understanding of the Phenomenon	Substantive Theories to Explain Specific Cases Cognitive Maps Causal Loop Diagrams Reference Modes Fully Developed System Dynamics Models with Dynamic Hypotheses Extended and Elaborated Generic Structures
Issues	Using Existing Literature in the Early Stages Being Free of Any Theoretical Preconceptions Theoretical Sensitivity and Generic Dynamic Structures Empirical Content of Theoretical Codes Falsifiability of Statements Heuristic Concepts	Nature of the Dataset Richness of the Dataset given the Purpose of the Research Length of the Data Analysis	Type of the Research Output Use of Generic Models/Archetypes
Solutions	Be Sensitive to Theoretical Bias Avoid High-Level Concepts Develop Heuristic Concepts	Evaluate the Dataset Broadly Focus on Understanding the System Use a Core Codebook	Every Finding is Important Ground your Dynamic Theory on the Data Report Gaps and Hypothesized Fillers