The use of the Case Study Method in Theory Testing: The Example of Steel eMarketplaces

Abstract

Many of the research questions of interest to IS academics and practitioners concern the success or failure of change initiatives involving the introduction of new systems and practices, when the phenomenon interacts with the context, and the focus is on organisational rather than technical issues. These are exactly the types of research questions for which a case study method is well suited. This paper assesses the use of the case study method to test hypotheses and build theory while investigating the phenomenon of steel emarketplaces. The paper draws upon the lead author's experience when working on her doctoral thesis 'Factors Affecting the Viability of Electronic Marketplaces: an Empirical Investigation into International Steel Trading'. Although the case research strategy has mostly been utilised for exploration and hypothesis generation, the case method is appropriate to all phases of research. In this study the research objectives were identified as theory description and theory testing, and the case strategy was used to describe and test the hypotheses. The lead author undertakes a cross-case analysis of multiple IT-powered initiatives in order to develop theoretical propositions to be tested through subsequent research. This paper discusses how issues and concerns inherent in this method were dealt with, and assesses the quality of the findings. [PUBLICATION ABSTRACT]

Headnote

Abstract: Many of the research questions of interest to IS academics and practitioners concern the success or failure of change initiatives involving the introduction of new systems and practices, when the phenomenon interacts with the context, and the focus is on organisational rather than technical issues. These are exactly the types of research questions for which a case study method is well suited. This paper assesses the use of the case study method to test hypotheses and build theory while investigating the phenomenon of steel emarketplaces. The paper draws upon the lead author's experience when working on her doctoral thesis 'Factors Affecting the Viability of Electronic Marketplaces: an Empirical Investigation into International Steel Trading'. Although the case research strategy has mostly been utilised for exploration and hypothesis generation, the case method is appropriate to all phases of research. In this study the research objectives were identified as theory description and theory testing, and the case strategy was used to describe and test the hypotheses. The lead author undertakes a cross-case analysis of multiple IT-powered initiatives in order to develop theoretical propositions to be tested through subsequent research. This paper discusses how issues and concerns inherent in this method were dealt with, and assesses the quality of the findings.

Keywords: case study research, positivist research, building theory from case studies

1. Introduction

Many of the research questions of interest to IS academics and practitioners concern the success or failure of change initiatives involving the introduction of new systems and practices, when the Information System interacts with the people using it, and the focus is on organisational rather than technical issues. These are exactly the types of research questions for which a case study method is well suited.

For Yin (1994) 'A case study is an empirical inquiry that investigates a contemporary phenomenon within its real-life context, especially when the boundaries between phenomenon and context are not clearly evident.'. In the case study attention is paid to contextual conditions, the focus is on contemporary events, and the experience of the actors is important.

The case study method is flexible, producing diverse research outcomes (Darke et al, 1998), and supporting all types of philosophical paradigms. Case studies can be exploratory, descriptive or explanatory (Yin, 1994). They can be used to generate and/or test theory within the positivist paradigm (Eisenhardt, 1989; Lee, 1989; Lee and Baskerville, 2003). They can be intrinsic, instrumental (providing insight into an issue or situation of concern) or collective - based on more than one site (Stake, 2000). They can be used to provide a rich description of social phenomena, generating knowledge of the particular within the interpretivist paradigm (Walsham, 1993; Macpherson et al, 2000).

This paper draws upon the lead author's experience of using the case study method when working on her doctoral thesis 'Factors Affecting the Viability of Electronic Marketplaces: an Empirical Investigation into International Steel Trading'. The research project is set against the background of the steel trading industry, in which the steel is bought and sold globally, and has to be transported physically. The phenomenon being investigated is the failed attempts between 1998 and 2000 at establishing electronic marketplaces to handle this trade. Economic theory and academic research predicted that electronic marketplaces would displace traditional intermediaries, the steel trading companies. These predictions failed to materialise. Hence, the research project initially focused on the questions - Why did these marketplaces fail' Under what conditions might they work' It later evolved into the revision of the existing theory and the testing of a revised theory. This paper reports on how the case method was used to build and test theory. The aim of the paper is to discuss how issues and concerns inherent in this method were dealt with, and to assess the quality of the findings.

The next section explains the rationale behind the choice of the method. Section three discusses how the case method can be used to test hypotheses and build theory within the positivist paradigm. The following section introduces the research project and discusses the research design, the data collection techniques, the data exposition and analysis. Section five evaluates the research process and the quality of the results.

2. Rationale behind the choice of the case method

Economic theory and academic research predicted that the advent of electronic marketplaces would revolutionise current business practice and lead to the disintermediation of the middleman (the Electronic Market Hypothesis - 'EMH'). The lead author's doctoral thesis looked for evidence in support of the EMH in the steel industry, where it was also predicted that electronic markets would replace the middleman, the steel trading companies. Many trading platforms sprang up between 1998 and 2000; however, none prospered. The research started with an in depth case of one failed attempt at launching an electronic marketplace (pilot case), and added more cases investigating different approaches to the same business problem.

The cases presented in the study conform to Yin's definition. They all investigate a contemporary phenomenon in a real-life setting, and the focus is on organisational and managerial (rather than technical) issues (Myers, 2003). These are precisely the issues identified as best suited to case study research. The experiences of the agents are critical, and intelligible only within the social and cultural context of the industry. No manipulation of variables is possible (or necessary), and there is still little cumulative knowledge in IS (Benbasat et al., 1987; Benbasat and Zmud, 1999).

Following Benbasat et al. (1987, p. 372), the lead author asked the following questions to decide on the suitability of the case strategy (the answers are provided in parenthesis):

* 'Can the phenomenon of interest be studied outside its natural setting'? (No)

* 'Must the study focus on contemporary events'? (Yes)

* 'Is control or manipulation of subjects or events necessary'? (or possible...' No; Benbasat et al., 1987)

* 'Does the phenomenon of interest enjoy an established theoretical base'? (No, as no cumulative knowledge in IS; Benbasat and Zmud, 1999).

The case study is the most common qualitative method used in information systems (Myers, 2003). Qualitative methods, such as ethnography, action research, case study research, were developed in the social sciences, and were deemed to be more appropriate to the study of social and cultural phenomena than the quantitative methods of the physical sciences, such as survey methods, laboratory experiments, mathematical modelling. Qualitative methods are concerned with the meaning, not the frequency, of phenomena. The rationale for conducting qualitative analysis is that, given the human capacity to talk, the object of understanding a phenomenon from the point of view of the actors is largely lost when textual data are quantified.

The use made of the case method in this study is that discussed by Eisenhardt (1989) and Lee (1989) - the cases are used to revise and extend an existing theory from practice, and then move on to the testing stage. Thus, the philosophical perspective adopted in this study is positivist. Positivism posits that reality is external and objective, and objectivity is both possible and desirable. Positivist research aims to produce universal laws through the application of the scientific method of the physical sciences (Lee and Baskerville, 2003) and to increase the predictive understanding of the phenomenon under investigation (Orlikowski and Baroudi, 1991). This was also the aim of the research - to revise the EMH and test the revised model.

3. The use of the case study method in building and testing theory

This section discusses the problems and challenges of case research for building and testing theory. It outlines a number of ways to evaluate the quality of the research findings.

Case study research is difficult and presents the researcher with unique challenges (Yin, 1994; Dubç and Parç, 2003; Darke et al, 1998). Case studies are most commonly qualitative, and this places a heavy emphasis on the individual contribution and choices of the researcher. The researcher seeks an in-depth understanding of the interaction between phenomenon and context. S/he may collect data on many variables; the challenge is to identify those variables of significance to the phenomenon under investigation (Benbasat et al, 1987; Remenyi & Williams, 1995). Hence the importance of methodological rigour to increase the robustness of the argument and the reliability of the findings. Key decisions for the researcher include the range and type of data to be collected, the selection and number of case sites and the type of analysis to be carried out.

Eisenhardt (1989) describes the process of building theories from case study research. The process starts with the initial definition of the research question. The 'a priori' identification of variables ('constructs') from the extant literature guides the research process. Tentative themes emerging from the fieldwork are compared and contrasted with the literature. The idea is to systematically compare and contrast theory and data, iterating towards a theory that accurately reflects the data. The comparison of emergent themes and theories with the literature is crucial, given the limited number of cases which can be studied.

Building theories from case studies relies on theoretical (as opposed to statistical) sampling. Given the limited number of cases which can be studied, it is important to select critical, extreme, revelatory cases, in which the phenomenon is 'transparently observable' (Pettigrew, 1988, cited in Eisenhardt, 1989). Inevitably an element of subjectivity is involved in non-random sampling. A multiple case design allows the findings to be replicated across cases; hence, the evidence from multiple cases is perceived as more compelling. A multiple-case design is appropriate when the purpose of the investigation is theory description, theory building or theory testing. The factors dictating the choice of the research design, whether single or multiple case, will guide the choice of site.

Within the positivist paradigm the case study can be viewed as an experiment. While it is not a randomised experiment, the logic in relation to generalisability is the same, and the investigator expects the same outcome from either the experiment or the case, provided that all the variables of interest are replicated. Cases may be selected for literal replication, for which the investigator expects similar outcomes, or for their differences with respect to the variables of interest, and for these the investigator expects contradictory outcomes (theoretical replication). Either way the reliability of the findings is enormously amplified when both types of cases form part of the research design.

There is now a de facto standard for evaluating case study research within the positivist tradition (Benbasat et al., 1987; Lee, 1989; Eisenhardt, 1989; Yin, 1994). This tradition acknowledges the importance of the three criteria of validity (internal for causality, external for generalisability), reliability, and replication. Dubç and Parç (2003) use these ideas to develop a framework for evaluating positivist case study research in information systems (Dubç and Parç's framework is referred to later in this paper).

The case study relies on multiple sources of evidence and multiple data collection methods. Each source has advantages and disadvantages and all complement each other, so that it is recommended that multiple sources of evidence be used and triangulated (Yin, 1994, p. 92). Triangulation enhances both validity and reliability. Within-case and cross-case analysis, cross-case pattern-matching and tying the emergent theory with the literature enhance internal and external validity. Internal validity is concerned with the issue of causality - do the factors claimed to drive the observed outcomes actually cause them or are they themselves caused by other factors' External validity refers to the generalisation of the findings beyond the case. Case studies are generalisable to theoretical propositions (analytical generalization) rather than populations, as the case study does not represent a 'sample', and the researcher's objective is to generalise theories rather than list frequencies. Reliability demonstrates that the procedure can be replicated with the same results, and is dealt with by making as many steps as explicit as possible, and clearly displaying the evidence, so that the process can be audited. The idea is to analyse the evidence objectively, eliminate alternative interpretations, and produce a compelling case. Data exposition and analysis rely heavily on 'the integrative powers of the researcher' (Benbasat et al., 1987, p. 374) and it is incumbent upon the researcher to present the evidence, establish cause and effect and argue persuasively. The reader should be able to follow the derivation of any evidence from the initial research question to the conclusions of the study, as this chain of evidence will improve the reliability of the findings (Yin, 1994).

Lee (1989) discusses four problems resulting from applying the natural science model to MIS case research. These are:

* Making controlled observations in a real-world setting. As no manipulation of variables is possible in natural settings, the researcher has to capitalise on naturally occurring variations in the variables.

* Making controlled deductions. Markus (1983) deals with the problem by making verbally expressed predictions based on different verbally expressed theories.

* Achieving replicability from real-world settings which rely on observations which are unique and non-replicable.

* Generalisability - how to extend and apply the finding from one case to other settings, given that the phenomena observed are unique and non-replicable.

The use of the case study method in building and testing theory has been extensively discussed and analysed in MIS literature. A consensus has formed that MIS research need not be quantitative in order to be regarded as scientific, and there is now an established methodology for conducting case study research within the positivist paradigm.

4. The research project

This section introduces the research project and discusses the research design and process.

The research project looked for evidence in support of the EMH in the steel industry, where it was also predicted that the advent of electronic marketplaces would lead to the disintermediation of the middlemen, the steel trading companies. The study focuses on the global physical Business-to-Business ('B2B') spot steel trading market. Spot market is defined as business which is not under (short or long-term) contract. The study is concerned with physical trade (not 'paper' trade = derivatives), and the material (if) bought 'virtually' has to be transported 'physically'.

An electronic marketplace allows its users to identify and select potential partners and conclude business online. Opinions differ whether an electronic marketplace needs to support all three functions of identification, selection and execution to qualify as an electronic marketplace. Suffice it to say here that electronic marketplaces vary considerably in the range of functions they provide; this is sometimes referred to as 'the scope of the electronic market' i.e. the range of functionalities/ancillary services offered, the degree of automation and geographical coverage. Many electronic marketplaces launched between 1998 and 2000 were merely 'quote engines', providing little more than aggregation/match-making. E-Steel and MetalSite were the first to announce plans to broaden their geographical reach and service offering. These plans never came to fruition. One by one the steel electronic marketplaces launched during the 'dot.com bubble' became inactive or were forced to change their business models and strategic focus.

4.1 The research design

In April 2000 the lead author became involved in a feasibility study for the development of an electronic marketplace targeted to serve the intercontinental steel trading market. This study stimulated the intellectual curiosity of the lead author; hence, the research project began with data rather than theory (Lampel, 2004). As a participant observer the lead author had unlimited access to relevant information throughout the duration of the project. She continued to work in the steel trading industry during the project; hence, the decisions made with respect to the case selection and research design flowed from her immediate working experience and access to industry professionals.

The research process started with the formulation of the research question. A review of the extant literature suggested a number of variables which might affect the viability of electronic marketplaces. The lead author organised the motifs which emerged in various papers in a coherent argument and tentative conceptual framework (the Model). She formulated the working hypotheses, and, based on the working hypotheses, made verbally expressed predictions to be tested in the fieldwork. Thus, the fieldwork features all the variables tentatively identified as determinants of e-marketplace viability. The Model is tested and refined through the fieldwork and recommended for further testing in other industry settings.

The study features multiple cases and multiple levels of analysis within a single case (firm vs. industry; prime vs. secondary market) and attention is paid to the relevant context; hence, each case has an embedded design. The author discusses the product characteristics and the structure of the industry, and the trade laws and business practices governing it, and introduces the viewpoint of buyers, sellers and traders.

The selection of cases relies on theoretical sampling. The cases were chosen from a larger sample because of their characteristics, and the variables of interest are clearly observable. The cases are selected for literal and/or theoretical replication, because of the intrinsic similarities and/or differences between them. The deciding factor, however, is accessibility.. In this sense, the selection of cases is partly opportunistic, and allows the lead author to rely on participant observation (Lee, 1989; Yin, 1994).

The final research design features seven cases. The experience of other electronic marketplaces is incorporated by reference in this study. The seven cases are presented in Table 1.

The first three cases investigate the prime market for steel products, in which the steel is made to order and to the specifications of the particular buyer. Cases four to seven investigate the secondary market for steel products, in which commodity items are typically offered on an 'as is' and 'where is' basis.

Good research design is an iterative, not a linear process (Lampel, 2004). The initial research design featured two cases, both representing attempts at launching an electronic marketplace allowing one-stop shopping in the prime market. Later it seemed appropriate to add a descriptive case at the beginning, which would serve as an introduction and provide the reader with an understanding of the dynamics of international steel trading. Thus, the research design was enhanced to feature three cases, all set against the background of the prime steel market.

The three cases are conceptually distinct and stand-alone, yet related. Whereas case one constitutes a reflection upon the business dynamics and processes of international steel trading, case two tests the hypothesis and also represents a meta-evaluation of the feasibility study in which the lead author had participated. Case three also tests the hypotheses, and the investigator expects a similar outcome (literal replication). The remainder of the fieldwork is concerned with the secondary market for steel products. The researcher can test the hypotheses in a different setting (the secondary market), and expect a contradictory outcome (theoretical replication). Two cases were chosen to introduce/describe the secondary market, the characteristics of transacted items, the role of the intermediary, and the scope of service. Later the opportunity arose for the lead author to utilise the services of two electronic marketplaces to dispose of reject parcels. She gained valuable insight into the workings of online auctions. The two electronic marketplaces became the subject of the last two cases.

4.2 Data collection

In this study multiple sources of evidence are utilised and triangulated. These include: company files, business plans, financials, published reports by management consultants, magazine and newspaper articles, slide-shows, emails etc. The lead author had informal exchanges with colleagues, senior executives of electronic marketplaces, journalists and logistics services providers mostly during the normal course of business. Hence, data were often collected in a non-systematic manner though informal personal communications. No formal interviews were conducted. The choice of data collection methods reflects the preference of the lead author for unobtrusive techniques, and a concern to minimise disruption and maintain a low profile. Inevitably this raises the issue of methodological rigour (e.g. none of the interviews were recorded; there are no transcripts). Field notes were taken, and revisited as soon as possible after the event to ensure accuracy, and filed electronically. Emerging issues were analysed and, where appropriate, further explored by telephone, email or instant messaging at the first opportunity. All contributors were informed of the scope of the study, and emerging themes were discussed with them. The prevailing data collection technique is participant observation (Myers, 1999); above all this thesis draws upon the experiences of the lead author in the industry.

4.3 Data analysis

In qualitative research data collection and analysis, interpretation and reporting are often carried on in parallel, and the results of one activity can alter the direction of the others. In this study the findings from the fieldwork are analysed through pattern-matching, within the case and across cases. Emerging themes are compared and contrasted with the literature and with the working hypotheses and predictions. Multiple theories are triangulated, alternative explanations are considered and the appropriate adjustments are made to the Model. The study features a sample of vignettes of practice, from real life situations, and photographs, to illustrate concepts and corroborate the argument. There is a checklist at end of each case to summarise the findings and monitor progress. The reader is able to follow the researcher's argument, but ultimately form his/her own opinion.

5. Evaluation of the study and research process

This section aims to assess the level of methodological rigour in the research process and its adherence to the criteria of good practice for positivist case study research in IS. Dubç and Parç (2003) propose a framework for evaluating positivist case study research in information systems. Tables 2 and 3 assess the quality of the research design and data collection and analysis against Dubç and Parç 's criteria. They show that the research process adhered to good practice and generally met the criteria of validity, reliability and replicability.

This study also conforms to Lee (1989). The four problems discussed in Lee (1989) are dealt with as follows (Table 4):

5.1 Quality of findings

Based on the foregoing, the refined Model is robust and can assist in identifying those purchasing situations which are promising for B2B electronic marketplaces; the characteristics for successful e-marketplaces are shown in sufficient detail for it to be used for further research in other industry settings. The parameters identified are generalisable to other industry settings and not steel specific.

In this study the starting point was a well-established theory, which the researcher's experience showed did not work. Subsequent case research clarified the objective factors which were needed to make it work. The robustness of the findings is due at least in part to the good fit of the positivist paradigm and research method with the scope of the enquiry. The academic work is enriched by the experience of the practitioner and the reliability of the findings is increased by the credibility of the researcher as an industry insider.

5.2 Lessons learnt

Research popular among steel industry practitioners between 1998 and 2000 predicting greater market governance was mainly quantitative. Predictions were said to be conservative, as one piece of steel could in principle be bought and sold over the internet several times. These studies failed to consider the highly dynamic, supply chain-intensive and relationship-filled trading environment. The industry knowledge is largely tacit, the level of personal involvement in the buying/selling process is high, and firms make decisions based on factors other than price alone. There is an argument here which supports qualitative vs. quantitative methods and 'inquiry from the inside' in management research, since the experiences of the actors are critical and intelligible only within the social and cultural context of the particular industry. The choice of the case method allows the writer to capitalise on unparalleled access to data and sites through participant observation. Here participant observation arises from an ongoing working situation. The mode of enquiry, therefore, fits the topic of the investigation and the investigator's skills and circumstances.

5.3 Limitations

Iacono et al. (2009) elaborate on case study research and the role of the researcher as a participant observer. The strength of this method is the in-depth and first-hand insight into a real world setting which the investigator acquires; one weakness is the potential lack of objectivity. The research project draws upon the lead author's experiences and the experiences of others in the industry; data have sometimes been collected in a non-systematic manner though informal personal communications during the course of business. Concerns over confidentiality have at times also prevented the writer from disclosing data of a sensitive nature, which might have further illustrated concepts and corroborated arguments.

In this study the lead author has attempted to present the evidence in an unbiased and clear manner. In the research project concerns over subjectivity and/or lack of rigour were dealt with by acknowledging the dual role of the investigator as an industry practitioner and researcher; distinguishing as appropriate facts from personal reflections, alternating between inside and outside enquiry, and documenting and substantiating statements and let the facts speak for themselves.

6. Conclusions

This paper has discussed the use of the case study method to build and test theory, and how issues and concerns arising from utilising this method were dealt with in the lead author's doctoral thesis.

The robustness of the findings is due to the nature of the phenomenon, which fits comfortably within the positivist paradigm, and the appropriateness of the case method, which is particularly suited to the study of information systems in organizations, and permits the collection of rich qualitative data through participant observation. This paper has shown that it is possible to produce rigorous qualitative research which complies with the standards of positivist research and meets the criteria of validity, reliability and replicability of the natural sciences.

References

References

Benbasat I, Goldstein D and Mead M (1987) 'The Case Research Strategy in Studies of Information Systems' MIS Quarterly Vol. 11, pp. 369-386.

Benbasat, I. and Zmud, R (1999) 'Empirical Research in Information Systems: The Practice of Relevance' in MIS Quarterly, Vol. 23, No. 1, pp. 3-16.

Darke P, Shanks G and Broadbent M (1998) 'Successfully completing case study research: combining rigour, relevance and pragmatism' Information Systems Journal 8, pp. 273-289.

Dubé, Line; Paré, Guy.(2003) 'Rigor in Information Systems Positivist Case Research: Current Practices, Trends, and Recommendations' MIS Quarterly,Vol. 27 Issue 4, p. 597.

Eisenhardt K (1989) 'Building Theories from Case Study Research' Academy of Management Review Vol.14, No 4, pp. 532-550.

Iacono J, Brown A and Holtham C, (2009) 'Research Methods - A Case Example of Participant Observation' Proceedings of the 8th European Conference in Research Methods for Business and Management (ECRM).

Lampel, J. (2004) 'A Note on Research Design' Working Paper, City University, Cass Business School.

Lee A (1989) 'A Scientific Methodology for MIS Case Studies' MIS Quarterly Vol. 13, Issue 1, pp. 33-50.

Lee A and Baskerville R (2003) 'Generalizing Generalizability in Information Systems Research' Information Systems Research Vol. 14, No. 3.

Macpherson I, Brooker R and Ainsworth P (2000) ' Case study in the contemporary world of research: using notions of purpose, place, process and products to develop some principles for practice' International Journal of Social Research Methodology Vol. 3, Issue 1.

Markus L (1983) ' Power, Politics and MIS implementation' Communications of the ACM Vol. 28, No 6, p. 430-444.

Myers, M.D. (2003) 'Qualitative Research in Information Systems' in MISQ Discovery, at www.qual.auckland.ac.nz.

Orlokowski W J and Baroudi JJ (1991) "Studying Information Technology in Organizations: Research Approaches and Assumptions", Information Systems Research (2) 1991, pp. 1-28.

Remenyi D and Williams B (1995) 'Some aspects of methodology for research in information systems' Journal of Information Technology 10, pp. 191-201.

Stake Robert (2000) 'Case Studies' Ch.16 of Denzin and Lincoln (ed) Handbook of Qualitative Research 2nd edition, Sage.

Walsham G (1993) 'Interpreting Information Systems in Organizations' Wiley Series on IS

Yin R (1994, 2003) 'Case Study Research Design and Methods' second and third edition Applied Social Research Methods Series, Volume 5, Sage Publications.