Experiences From Sequential Use of Mixed Methods

Abstract

The discussion of qualitative or quantitative approaches has been going on for many years. One way to reduce the most dogmatic standings is to use mixed methods consisting of combinations of qualitative and quantitative approaches. In this paper, we have analysed usage experiences from combining qualitative and quantitative approaches in different ways. We refer to these combinations as method configurations. Our findings point out that a researcher should commence with a qualitative approach when: 1) the researcher has a lower pre-knowledge of phenomenon to be studied, 2) the phenomenon to be studied is abstract and 3) there is an uncertainty if the questions asked are the right questions. On the contrary, there is a tendency in our results that the researcher should start with a quantitative study when 1) the researcher has a good pre-knowledge of the phenomenon or 2) the phenomenon is more concrete. [PUBLICATION ABSTRACT]

Headnote

Abstract: The discussion of qualitative or quantitative approaches has been going on for many years. One way to reduce the most dogmatic standings is to use mixed methods consisting of combinations of qualitative and quantitative approaches. In this paper, we have analysed usage experiences from combining qualitative and quantitative approaches in different ways. We refer to these combinations as method configurations. Our findings point out that a researcher should commence with a qualitative approach when: 1) the researcher has a lower pre-knowledge of phenomenon to be studied, 2) the phenomenon to be studied is abstract and 3) there is an uncertainty if the questions asked are the right questions. On the contrary, there is a tendency in our results that the researcher should start with a quantitative study when 1) the researcher has a good pre-knowledge of the phenomenon or 2) the phenomenon is more concrete.

Keywords: mixed methods, method combinations, mixed approaches, qualitative methods, quantitative methods

1. Introduction

Over the years there has been an on-going debate regarding qualitative approaches and quantitative approaches. Often this debate has been controversial (cf. Bryman, 1988). Followers of quantitative approaches are often characterised as positivists, while the followers of qualitative approaches are often characterized as hermeneutists. Followers of quantitative approaches often criticise qualitative approaches for: 1) results are hard or impossible to generalise (ibid), 2) results are not objective and hard to replicate and 3) there is a lack of transparency (Bryman, 2009). We agree with this criticism since qualitative studies are often based on one or a few case studies and many times data can be hard to interpret.

Vice versa, followers of qualitative approaches criticise quantitative approaches for 1) using irrelevant hypothesis (Blumer, 1956) and 2) descriptions are too superficial (Schutz, 1962). We concur with this criticism. In our opinion, hypothesis stemming from qualitative research are more grounded than hypothesis stemming from quantitative research. We also agree that the qualitative approaches have a better opportunity to achieve "richer" descriptions. One basic difference between these two approaches is that the aim of a quantitative approach is to suggest a hypothesis that should be verified or falsified while the aim of a qualitative approach is to generate a hypothesis. Simplified, it seems as the primary interest of the qualitative approaches is to justify "Is this the right question to ask?" while the quantitative approaches are more concerned with "Is this answer trustworthy?". Our view of research is that every approach has advantages and disadvantages. This is also valid for qualitative and quantitative approaches.

Many scholars have brought forward the idea of combining qualitative and quantitative approaches (e.g. Bryman 1988, Bryman, 2009, Creswell, 2009). The objective of combining the two approaches is to preserve the strengths and reduce the weaknesses in both approaches (Bergman 2009b). The preferred term for combining these approaches is "mixed methods" (Bryman, 2009). The aim of this paper is to present usage experiences in terms of strengths and problems from combinations of qualitative and quantitative approaches. According to Bergman (2009a) mixed methods have experienced a tremendous increase in popularity. Creswell (2009), identifies that there is not much written about experiences from certain types of combinations of qualitative and quantitative approaches which hamper the practical use of mixed methods in research. In this paper, we are presenting experiences from real use of combinations of qualitative and quantitative approaches. This introductory section is followed by a description of our theoretical bases and in section 3 we outline our research approach. In section 4, we present our findings. Finally, in section 5 the conclusions are presented.

2. Theoretical basis

According to Creswell (2009) a mixed method is more than simply collecting and analyzing data from a qualitative and quantitative approach. The mixed method approach provides a specific perspective of the world. Besides, it also incorporates a combined qualitative and a quantitative approach. That is, the overall strength should be more beneficial than using qualitative or quantitative research individually. One aim of using a mixed method is to increase the possibility to achieve findings that are more trustworthy and relevant than using the approaches separately.

A key design component in mixed method research is whether the research methods are implemented in a parallel manner or in a sequential manner (Teddlie & Tashakkori 2009). Sequential mixed method research refers to an investigation in which the phases of the research occur in a consecutive order, with one phase emerging from or are following the other. The research questions addressed as well as the procedures used in one phase depend on the previous phase. In parallel mixed method research the research includes phases that occur either simultaneously or with some time lapse. These phases address related parts of the same research questions.

Greene & Caracelli (1997) presents another typology how mixed methods may be implemented. It includes two broad classes of designs (component, integrated), with a total of seven separate forms of mixed methods: component (triangulation, complementarity, expansion) and integrated design (iterative, embedded or nested, holistic, transformative). Based on Greene & Caracelli (1997) a brief description of these designs follows. In component mixed method research the data collection procedures are implemented as separate aspects and remain distinct throughout the research. Findings derived from one method are in triangulated mixed methods research used to corroborate findings generated with other methods. In complementary research findings from one dominant method are strengthen and improved through findings from another method. In expansive mixed method research different methods are implemented to generate results for separate parts of the study; results are presented "side-by-side" In integrated mixed method research the methods used are integrated throughout the evaluation. In iterative research this means that a dynamic interplay of findings has been developed through the use of different methods throughout the evaluation stage of the study. In embedded or nested mixed method research one method is utilized and "located" within another method in order to stimulate a creative tension during the study. Holistic integrated mixed method research means the simultaneous integration of methods throughout the study, building towards one integrated explanation of results. In transformative mixed method research the methods are used to capture differing value commitments in order to facilitate transformation (Greene & Caracelli 1997).

Johnson & Onwuegbuzie (2004) present strengths and weaknesses of mixed method research. As strengths, they point out that words, pictures and narratives can be used to add meaning to numbers; i.e. how qualitative research could facilitate 1) the analysis of quantitative data, and 2) the design of hypothesis, scales and indices for quantitative research (Bryman 1988). Vice versa, numbers could be used to add precision to words, pictures and narratives; i.e. how quantitative research could facilitate qualitative research (Johnson & Onwuegbuzie 2004) such as verifying or falsifying an empirical grounded hypothesis. Furthermore, a mixed method approach can also manage a broader and more complete range of research questions because the researcher is not solely confined to a single research approach or method. The use of a mixed method can also provide stronger evidence for a conclusion through convergence and corroboration of findings. By combining qualitative and quantitative approaches, or triangulate, the researcher's claim for validity of his or her conclusions are enhanced if they could be shown to provide mutual confirmation (Bryman 1988). Mixed methods can also add insights and understanding that might be missed when only a single method is used (Johnson & Onwuegbuzie, 2004) and a combination of methods could also increase the ability to generalise the results compared to a qualitative study (Bryman 1998). Qualitative and quantitative approaches used together produce more complete knowledge necessary to inform theory and practice (Johnson & Onwuegbuzie 2004).

Johnson & Onwuegbuzie (2004) also point out several weaknesses regarding mixed methods. One weakness is that it could be difficult for one researcher alone to carry out both qualitative and quantitative research. This especially is the case if two should be used concurrently. A design embracing concurrency might require a research team. Concurrency involves more participants and more activities, which calls for more expenses. Moreover, a concurrent design is time consuming. Other barriers to the mixing of methods is that the researcher has to learn about multiple methods and their internal logic in order to mix them accordantly, defend the use of them and also be able to use them in a professional manner. It is often more simple to focus on a single method or approach. Another weakness pointed out both by Johnson & Onwuegbuzie (2004) and Bryman (1988) is that methodological purists contend that a researcher always should work within either a qualitative or a quantitative paradigm and not mix the two.

According to Bryman (2009), there are at least two arguments against mixed methods. The first argument is that research methods carry epistemological commitments and the second argument is that the two approaches represent separate paradigms. The argument concerning epistemological commitments is based on that every research method is embedded in a specific way of perceiving the world. That is, to use a questionnaire, to be an observer or to measure something is to be involved in conceptions of the world which allow these methods to be used for their purposes (Hughes, 1990). The argument concerning separate paradigms views qualitative and quantitative research methods are incompatible (Guba, 1985; Morgan 1998).

We respect these views but we do not agree with the understanding that these two approaches represent two different "worlds" that cannot be integrated. Our view is supported by Teddlie & Tashakkori (2009). They claim that mixed method is an alternative to the dichotomy of qualitative and quantitative approaches. Of course, we realise that this integration could not be done in a naive way resulting in eclectically obscurities. Our belief is that there exist knowledge that could be shared between the "worlds"; but the sharing has to be carefully done in order to reduce undesired connotations that could emerge from an unreflect transfer of concepts from one specific context to another context. It is important that the original meaning of concepts is preserved. We definitely believe that both "worlds" would benefit from integration. The researcher has to be aware of that different underlying worldviews exist. Instead of viewing the two approaches as an either-or-position we believe it is more productive to perceive them as complements. They can support each other either as a sequential or as a parallel process. Another similarity is that both approaches are interested in answering how- and why-questions (Casebeer & Verhoef, 1997)

3. Research approach

The aim of this paper is to present experiences based on real use of sequential combinations of qualitative and quantitative approaches. In order to present experiences based on sequential combinations we have used two knowledge sources; empirical data and theory. The empirical data sources have consisted of two types of sequential use. The configuration of the first type commenced with a qualitative method followed by a quantitative method. The configuration of the second type commenced with a quantitative method followed by a qualitative method. The reason for analyzing both these configuration types is that we wanted to understand possible differences related to order. Both configuration types consisted of two cases (= four cases in total). These four cases represent three bachelor theses and one PhD thesis. In order to support our analysis we have used existing theory about mixed theory (see section 2). The four cases and existing theory have formed a based for presenting experiences (see figure 1).

We have analysed the four cases sequentially. An introductory analysis of the four cases revealed a pattern; they had a common generic research process consisting of five similar components. These components are: A) aim and description of the study, B) research design considerations, C) research part 1, D) research part 2 and 5) final results. In order to collect experiences, we decided to use these components as analysis units (see figure 2). The first component "Aim and description has been analysed for understanding the context of the study, the second component "Research design considerations" has been analysed for findings arguments for planning the study in a specific way, the interface between the third and fourth component "Research part 1" and "Research part 2" has been analysed in order to identify problems and strengths that can appear in the interface between the two research parts. That is, experiences from the first research part can be seen as conditions that may affect the second research part. We have not been looking for problems or strengths that uniquely can be related to one of the approaches. At last, we have analysed the final results. In particular, we have analysed the results generalisabilty, if the result can be traced back to the empirical data and if the two research parts complemented each other

Experiences have been gathered by interviewing the scholars that have carried out the four cases. In order to avoid confusion we refer to these scholars as informants and we use the term respondent when we refer to the people the informants have interviewed. The interviews consisted of semi-structured questions (Patton, 1990). In order to better understand the informants' experiences we have asked questions such as: "what is the pre-knowledge of the informants?", "can the studied phenomenon be characterised as abstract or concrete?", "what is the character of the output from the first part and how can the following part benefit from this?", "is there a need for adjustment before entering the second part or is it a straight forward process?", "can any type of data be transferred from part 1 to part 2 or are there any limitations?". The role of the theory (literature) has been to explain and complement the experiences.

4. Findings

4.1 Configuration type 1 (part one qualitative, part two quantitative)

4.1.1 Case 1A

Aim and description: In the first case (case 1A) the informants has investigated how a new idea was transferred from the original innovators, via requirements specification, through systems development and finally to implementation. Before the study, the informants had a low pre-knowledge of the studied phenomenon and they were not experts on the research methods used. The study commenced with face-to-face interviews based on semi-structured interview questions. The results from the interviews were followed up by a broad survey.

Research design considerations: The informants utilized a sequential mixed method design in which the qualitative and quantitative parts were viewed as separate components and used to improve the result in steps, hence similar to a triangulated design (Greene & Caracelli, 1997). The reason for the selected order is that the informants considered it as being too hard to commence with a quantitative study since the studied phenomenon was perceived as being too complicated. The categories used in the succeeding quantitative part would probably not have been identified without a preceding qualitative study.

Problems and strengths in the interface between the two approaches: An overall experienced strength is that all the categories identified in the qualitative part could be tested in the quantitative part. However, the transfer to the quantitative part was not a straightforward process. Some categories in the qualitative part were too widely formulated. These categories had to be reformulated into more concrete sub-questions since the answers to the questions in the quantitative part consisted of mutually exclusive choices. Another problem in the qualitative part was that sometimes there existed case-specific concepts used. These concepts were understandable within the studied organisation but they would probably not be understandable in other contexts. That is, the informants had to find more abstract concepts before sending the questionnaire to a larger population. The informants perceived this process as hard since it was important to preserve the original meaning of the concepts.

The quality of the final results: It is possible to trace the final results all the way back to the respondents' utterances. That is, the traceability from empirical data to conclusions is high which also supports the credibility of the results. However, necessary specifications of concepts have been made along the process. These specifications have contributed to a precision in the final results. Furthermore, thanks to the succeeding quantitative part, there is no doubt that the final results are easier to generalise.

4.1.2 Case 1B

Aim and description: The second case (case 1B) analysed whether available web based information resources for higher education are suitable and easy to use (e.g. structure and content). User behaviour concerning information search was especially studied. One aim was to identify young persons' information behaviour related to existing functionality of a web-based e-service. The informant had a high pre-knowledge concerning qualitative methods but a lower pre-knowledge about quantitative methods. The study started with a qualitative text analysis and the aim was generate categories. This part was succeeded by a questionnaire that was sent to first year students and to last year students at upper secondary school. A pilot study was carried out before sending out the final questionnaire.

Research design considerations: The idea of using an introductory text analysis was to get an initial understanding of the problem area and to get an understanding of what categories are more interesting to study in a succeeding quantitative analysis. That is, the study was design as 1) an introductory open and broad approach, 2) a selection of interesting categories was made and 3) more information was selected regarding the selected categories. The main idea of a succeeding quantitative study to confirm initially generated categories making the mixed method research design applied in this study similar the complementarity mixed method design (Greene & Caracelli, 1997).

Problems and strengths in the interface between the two approaches: The output from the qualitative text analysis consisted of a list of categories that were considered for further analysis. The aim of the succeeding quantitative analysis was to get a deeper knowledge about the categories. A perceived strength is that the introductory text analysis worked as a good base for formulating the quantitative questions. The informant did not perceive any problem in using the categories as a base for formulating quantitative questions. The informant mentioned the problem of avoiding formulation of leading questions. According to the informant, questions consisting of valuations are to a certain extent always leading. The informant thought that it would be possible to collect the respondents' values through less leading questions; but to proceed with less leading questions would be time consuming. One idea of using a sequential mixed method is to be able to formulate more specific questions in "part 2". We agree that formulation of "leading" questions should be avoided. However, specific formulations that are based on experiences from "part 1" and should not be confused with "leading" questions. To be able to formulate specific questions is an opportunity provided in sequential mixed methods.

The quality of the final results: The quantitative and the qualitative questions complemented each other since they illuminated different perspectives of the problem area. One example is that the quantitative analysis provided a nuanced picture of how important the applicants perceived each category. That is, new attributes were identified in the succeeding quantitative part.

The qualitative analysis also informed about similarities and differences between groups of applicants. The combination of quantitative and the qualitative questions contributed to a clearer picture of wholeness. According to the informant, the results provided a good understanding of the problem area. Despite that the informant has carried out a combined qualitative and a quantitative study, she is very careful regarding the ability to generalize the results.

4.2 Configuration type 2 (part one quantitative - part two qualitative)

4.2.1 Case 2A

Aim and description: The third case included a study of the usability of computer support in systems development. The aim of the study was to suggest attributes of the computer support in order to improve the usability. The study embraced three steps: 1) a survey, 2) interviews and 3) observations. The informant considered himself as being very comfortable with qualitative methods and he had also a good idea of how to use quantitative methods.

Research design considerations: A condition of this study was that the studied phenomenon was not well researched. Only a few scientific studies existed which meant that there was not much knowledge that could be used as a platform for the study. Thus, the informant decided to commence with a broad survey. The aim of this survey was to get an overview, to become familiar with the studied area and to identify common existing problems. This survey was followed up by interviews in order to get a deepened knowledge. A selection of the most common identified problems from the survey was made. Finally, as a third step the informant used observations. The aim of these observations was to further deepen the study. Again, a selection of observations was made among the respondents. In this way, the research design can be pictured as a funnel where the interesting data from the introductory survey were selected and further deepened in the continuing expanding steps.

Problems and strengths in the interface between the two approaches: The informant experienced several strengths. One of the strengths was that uncertainties concerning interpretations from the introductory study could be reduced in the following interviews and observations. Another strength was that the identified problems in a natural way could be transferred to questions in the interview. Before the interviews were carried out similar problems had to be categorised in order to form suitable question areas. Not every problem was transferred to the interview for a deeper understanding; a selection was made. The selection criteria used was frequency (many respondents expressed the same problem) and articulation (fewer respondents articulated the problem well). The interview respondents were selected among the respondents of the survey. The selection criteria used in this case were: engagement in the survey, complexity in the information systems project carried out, the size of the information system project and the respondents own practical experience from use of the computer based tools studied. In the same way, users to observe were selected among the respondents that were interviewed. A third experience in the interface between the interviews and the observations was that new categories were identified which showed that the original problem categorisation was invalid. A new structure of categories had to be made according to the new findings. The third step (observations), can thus be viewed as quality insurance procedure.

The foremost problem was to interpret answers from the survey. An over-interpretation could lead to a wrongly drawn conclusion as well as an under-interpretation could lead to an unnecessary weak conclusion. Thus, the informant was happy that the research design embraced opportunities to reduce uncertainties in the interpretation process. The informant claimed that the degree of precision could be lost if not more detailed questions were asked.

The quality of the final results: The quality of the final result is high since the data collection embraced three distinct steps with three separated data analysis. Clearly, the description of the problems discovered in the first step was richer in the second and third step. Thus, the knowledge about the studied phenomenon was successively deepened. New data was discovered along the study that affected the initial formulation of the research question. The research question was adjusted and refined according to new insights. Attempts to claim generalisation was made. The informant used the concept of "generalisation levels" and stressed that the findings should be valid in other context with the conditions.

4.2.2 Case 2B

Aim and description: The fourth case embraced an investigation of a web based text design on small screen devices (7-9 inches PC). The first quantitative part was designed as an experiment consisting of 1) texts that were read and 2) a questionnaire using the Likert scale. The informant has examined how variables such as length of a row and font size affect readability and legibility. A statistical test was used to measure the respondents' answers in order to identify significant relationships between reading speed and how the text was designed. The quantitative part was followed up by a qualitative part consisting of interviews. The aim of the interviews was to further investigate preferences in relation to the variables used. An interview protocol was designed enabling semi-structured data collection. The interview respondents were selected among the respondents of the questionnaire. The informant considered herself to have a high pre-knowledge about quantitative methods but felt pretty uncomfortable in qualitative methods.

Research design considerations: The informant's knowledge about the studied phenomenon was considered as high. Therefore, she decided to start with an experiment consisting of well-developed hypotheses. Another consideration was that the informant chose not to plan the succeeding interviews in detail. The reason was that the informant's experience of performing interviews was low. The informant's attitude towards qualitative studies was also a bit sceptical; still the informant realised that her study would benefit from a mixed method design and she consequently applied a complementarity design.

Problems and strengths in the interface between the two approaches: A strength experienced was that the quantitative study provided a good base for the succeeding interviews. Several of themes (hypotheses) from the quantitative part were transferred to the semi-structured interviews. The informant did not experience any particular problems in the interface between the quantitative and qualitative part. A problem not specifically addressed was that there were no explicit criteria used for which themes to transfer. A subjective selection was made.

The quality of the final results: It is possible to trace the final results all the way back to the questionnaire. The qualitative study was merely used to confirm some of the findings in the quantitative study. That is, the full potential of the qualitative study was not utilised. Besides, the qualitative study provided some insights in how the respondents experienced the quantitative questionnaire. That is, the informant got feedback concerning how the experiment was interpreted and could use this feedback in relation to interpreting the final results.

4.3 Comparison of configuration type 1 and configuration type 2

The model we have used for comparing the two configuration types are pictured in figure 3. The picture consists of a generic situation including a researcher using a tool (a method) in order to study a phenomenon.

Based on this model we have analysed the cases in order to understand similarities and differences (see Table 1). The categories presented in the first column to the left are induced from the findings.

Based on the information in table 1, there are at least two tendencies;

* Informants having a lower pre-knowledge of the studied phenomenon or the studied phenomenon is more abstract then it might be a good idea to use configuration type 1.

* Informants having a higher pre-knowledge of the studied phenomenon or when the character of the phenomenon is more concrete then it might be a good idea to select configuration type 2.

We have also compared experienced problems and strengths in the interfaces between part 1 and part 2 regarding the two configuration types (see figure 2). One result regarding configuration type 1 is that the generated categories from the qualitative approach provided a good base for a succeeding quantitative approach but the process of transferring the generated categories from the qualitative approach was not a routine-like process and could not be done automatically. This process needed reflection since the concepts used in the categories were sometimes too wide or too contextual in order to fit in the succeeding qualitative part.

The major strengths regarding configuration type 2 are that: the quantitative study provided a good base for the succeeding interviews, uncertainties concerning interpretations from the quantitative approach could be reduced in the following qualitative study and interesting answers from respondents could easily be deepened in the succeeding interviews and observations. A succeeding deepened qualitative study meant that a selection of interesting areas was chosen. The possibility of making a selection allows the researcher to modify or refine the research question during the study. The major problem in configuration type 2 is the same as for configuration type 1; the results from the quantitative part could not automatically be transferred to the qualitative part.

In the final results, one strength for configuration type 1 is that it was possible to trace the final results all the way back to the informants' utterances. That is, the traceability is high. Not surprisingly, the final results are also easier to generalise when a succeeding quantitative approach was performed. The traceability is also considered to be high for configuration type 2. However, the generalisation ability is considered to be lower since the succeeding qualitative approach embraced fewer respondents.

5. Conclusions

The knowledge contribution of this study is experiences from use of mixed methods. We can conclude that the users' experiences concerning both the configuration types embraced similar and different strengths and problems (see section 5.3). One way to interpret the results is to use them as a guide for a research design of a mixed method. In summary, the tendency in our results shows that the researcher should commence with a qualitative approach when: the researcher has a low pre-knowledge of phenomenon to be studied, when the phenomenon is more abstract and when there is an uncertainty if the questions asked are the right questions. On the contrary, there is a tendency in our results that the researcher should start with a quantitative study when the researcher has a good pre-knowledge of the phenomenon or when the phenomenon is more concrete.

According to Creswell & Plano Clarce (2011), there are three categories that are important to consider in relation to mixed methods. These categories are: time and resources, pre-knowledge of the methods and the question of convincing researchers of the value with mixed methods. These categories can be viewed as arguments for or against using mixed methods. Besides Creswell & Plano Clarces' categories we have identified the two categories "pre-knowledge of the phenomenon to be studied" and "character of the phenomenon to be studied". These categories do not represent arguments for or against using a mixed method; rather they represent arguments for how to design (configure) a mixed method.

In section 2, we challenged the view that qualitative and quantitative approaches are incompatible. Our findings show that the informants have mixed the qualitative and quantitative approaches in a pragmatic way. They have not been obstructed by a worldview consisting of two approaches representing totally separated paradigms (cf. Kuhn, 1970). Mixed methods are not a technique; it is an attitude (Fielding, 2009). It is a conscious approach to get research quality and adequate explanations of social phenomena.

Our results are based on four cases; thus we cannot claim that they are valid for every research design situation. However, we do think that the results could work as inspiration for research design considerations of mixed methods.

References

References

Bergman, M. M. (2009a) "Introduction: Wither Mixed Methods"; Advances in Mixed methods Research, Sage Thousands Oaks.

Bergman, M. M. (2009b) "The Straw Men of the Qualitative-Quantitative Divide and their Influence on Mixed Methods"; Advances in Mixed methods Research, Sage Thousands Oaks.

Blumer H. (1956) "Sociological Analysis and the "Variable"", American Sociological Review, 19:3-10

Bryman, A. (1988) Quantity and Quality in Social Resarch Routledge, New York

Bryman, A. (2009) Social Research Methods 3rd edition Oxford University Press, Oxford

Casebeer, A. L. (1997) "Combining Qualitative and Quantitative Research Methods"; Chronic Diseases in Canada, Vol 18, No 3.

Creswell, J. W. (2009) Research Design: Qualitative, Quantitative, and Mixed Methods Approaches Sage, Thousand Oakes

Creswell, J. W. & Plano Clarce, V. I. (2011) Designing and Conducting Mixed Methods Research Sage, Thousand Oaks

Fielding, N. (2009) "Analytic Density, Postmodernism, and Applied Multiple Method Research"; Advances in Mixed methods Research, Sage Thousands Oaks.

Greene, J. C. Caracelli, V, J. (1997) Defining and describing the paradigm issue in mixed-method evaluation. In Greene, J. C. Caracelli, V, J. (Eds.) New directions for evaluation: Number 74: Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. Jossey-Bass, San Francisco.

Guba, E. G. (1985) "The Contect of Emergent Paradigm Research" in Lincoln Y. S. (ed.): Organization Theory and Inquiry: The Paradigm Revolution Sage, Thousand Oakes

Hughes, J. A. (1990) The Philosophy of Social Research 2nd edition Longman, Harlow

Johnson, R. B. & Onwuegbuzie, A. J. (2004): "Mixed Methods Research: A Research Paradigm whose Time has Come" Educational Researcher, 33(7), 14-26S

Kuhn T.S. (1970) The Structure of Scientific Revolutions, 2 ed. University of Chicago Press, Chicago.

Morgan, D. L. (1998) "Practical Strategies for Combining Qualitative and Quantitative Methods: Applications for Health Research" Qualitative Health Research 8:362-76

Patton M Q (1990), Qualitative Evaluation and Research Methods, second edition, Sage PublicationsSchutz, A. (1962) Collected Papers I: The Problem of Social Reality Martinus Nijhof, The Hague.

Teddlie, C. & Tashakkori, A. (2009) Foundations of Mixed Methods Research, Sage, Thousands Oaks.