How we Ensured Rigor from a Multi-site, Multi-discipline, Multi-researcher Study

Abstract

Qualitative research has often been criticised for its lack of rigour. In order to overcome this, measures of trustworthiness, dependability and reliability have been suggested. A study of how pastoralists learn to incorporate sustainable farming systems in the tropical savannas of Australia employed multiple-researchers, working in three States and from a variety of disciplines. To ensure rigour a framework for the study was developed by the researchers prior to commencing interviews. This was followed by regular teleconferences to ensure that the framework was valid and to adjust for any problems encountered along the way. Every interview was analysed independently by all researchers before a workshop was conducted to bring the ideas together. Categories and ideas within the data were synthesised to create an overall understanding of the learning process within the confines of "landcare" in the Tropical Savannas. These processes were undertaken in consultation with the pastoralists and the process has been explicitly documented to enable readers to follow the research process easily. The rigour in this project is shown in the clear documentation of the research process carried out by individual researchers and by the team when it met. The understanding of pastoralists' learning processes is our interpretation; it is up to the reader to decide whether s/he agrees with that interpretation, but from the description of the process it is easy for the reader to see where and why her/his interpretation differs.

URN: urn:nbn:de:0114-fqs0001125

1. Introduction

The purpose of the research project portrayed in this paper was to identify, describe and share knowledge and understanding of the learning processes of pastoralists in the tropical savannas of Australia. It was our intention that by better understanding and mapping these learning processes and activities, the research would further inform the practices of education and service providers, expand the effectiveness of communication between pastoralists and others, and empower pastoralists' to manage change more easily themselves and support others to manage both change and information. [1]

The research was supported by the Cooperative Research Centre for the Sustainable Development of Tropical Savannas (TSCRC) The TSCRC was established in 1995 under the Commonwealth of Australia Government's CRC Program and is now one of 67 in operation throughout Australia. Its focus is the sustainable development of the tropical savannas that cover almost one-quarter of the Australian continent. Achieving sustainable land management across the tropical savannas involves understanding how the various savanna ecosystems function. The TSCRC brings together researchers from different disciplines, sectors and regions who work with the land users to find practical solutions to the region's management issues. Educational courses on tropical environmental management that meets the needs of a range of people in urban and remote areas have been developed to extend this research to the whole region. [2]

Using a phenomenological approach, in-depth interviews explored how "pastoral families" were stimulated to learn, how they went about learning and how they gained and developed knowledge. They concentrated, in particular, on learning processes concerned with making changes to station or Rangeland management practices. These ranged from the simple changing of mustering of cattle to the more complex modification of fire regimes, introducing cell (a form of rotational) grazing or diversifying their sources of income. [3]

It was evident at the start of the project that a process would be required to ensure rigour in the research, particularly as field researchers from different disciplines, backgrounds and from the three States (Western Australia, Northern Territory and Queensland), were involved. The research framework was developed at the same time as the study's objectives (see research process). [4]

Many authors have debated the issue of rigour in qualitative research. This paper demonstrates how we maintained rigour in a multi-site, multi-disciplinary, multi-researcher study, particularly as defined by LINCOLN and GUBA (1989). In addition, we suggest that the rigour of our study is strengthened by our explicit reporting of how we achieved what we did and that this is a crucial element of maintaining rigour within qualitative research irrespective of the methodology you might be using. [5]

2. Our Research Process

Field researchers were selected for each of the three States and were invited to participate in a three-day workshop in Darwin, in the Northern Territory of Australia. During the workshop the broad objectives of the research project were developed as well as the framework, style and research method, including a "checklist" of questions to be used by each researcher. A project proposal was further developed at this time. [6]

The researchers had either worked with and knew the pastoral family to be interviewed or had had experience in the type of research methodology proposed for collecting the data. This helped to develop the project and to remove many barriers of mistrust, permitting more in-depth interviews and effective gathering of rich data from the pastoral families. It did, however, lead to some concern over the efficacy of some of the data collected and special processes were put in place during the analysis phase to deal with this (see discussion below). [7]

Field researchers conducted a "trial" interview before the main study. This was followed by a tele-conference to discuss strengths and weaknesses in the techniques and to check if there were any differences in the application of the research method being used. This allowed field researchers to more closely use the same techniques during the research process that followed. [8]

Prior to the formal interview, each pastoral family was contacted to gain their interest in the project and to build rapport (see CRAWFORD 1996; GARDNER et al. 1997; LEYBOURNE 1997). Eighteen semi-structured, in-depth interviews were held with pastoral families in Western Australia, The Northern Territory and Queensland. At each interview, specific learning episodes were identified and the discussion revolved around these to collect in-depth data about each episode. The episodes were not identified prior to the interview; it depended on issues raised by the pastoralists during the interview to determine which direction it took. [9]

A second tele-conference was held part way through the interviewing period, which allowed an opportunity for the researchers to bring up any issues or problems they were having. [10]

Following each interview, and before it was analysed, the transcript was sent to each family for comment. Contact was maintained with the pastoralists during, and after, the data analysis process to allow for their continued input into the project outcomes. [11]

Once pastoralists had agreed that the transcript was an accurate record of each conversation, a full set of the transcripts was sent to each member of the research team for coding prior to analysis. Several tele-conferences were held during the analysis phase to discuss important issues raised in the transcripts and to ensure that each person was coding the data in a complementary, if not similar, fashion. All the tele-conferences were recorded and transcribed. The codes assigned by individual team members were not pre-arranged and each researcher was asked to develop their own set of codes for the data. The only suggestion made was that overarching themes should be identified as a starting point for more detailed analysis at a later date. [12]

A second workshop was to bring the whole research team and the individual coding and analyses together. The research team identified three major themes: "perceptions", "informal learning" and "sources of information". Each of these three themes was further subdivided into sub-themes. For example, "informal learning" was subdivided into "observation", "experience", "trial and error", "talking/listening/discussion", "questioning", "reflection", "transfer of knowledge/principles", "networking/mentoring", "transposing positions", "information exposure" in terms of positions the pastoralists may have held, "groups", "opportunistic/timing" and "triggers". [13]

To ensure cohesiveness among the group and to remove any suspect data (see earlier comment), researchers worked in pairs highlighting data from each interview that related to each of the sub-themes. These pieces of data were then appended under each sub theme heading to begin to create the "rich picture" of what we were studying. To help with documenting the analysis process, the discussion during the workshop was recorded and transcribed in order to capture other issues raised. [14]

To enhance the contextual and phenomenological nature of the research a number of stories were identified from the transcripts to highlight the learning processes described by the pastoral families. The stories were developed following the workshop. These assisted individual researchers to look individually at each theme and ensure that all issues raised in the interviews were correctly covered by the workshop. These were discussed further in another tele-conference. [15]

Finally four members of the research team spent four days together to start writing the research report. During this stage the data collected for each theme was further consolidated to help focus the writing team members on the issues and to develop cohesive research outcomes. [16]

3. Discussion, and how we Ensured Rigour

The project used a phenomenological approach, which is a focus on understanding the meaning events have for persons being studied (PATTON, 1991). This is, by nature, qualitative research within the constructivist inquiry paradigm, as it looks to understanding a situation as the participants construct it. It sets out to capture what people say and do and how they interpret the world. As researchers, we need to be able to capture this process of interpretation through an empathic understanding or the ability to reproduce the feeling, motives and thoughts behind the actions of others, while at the same time remaining as objective as possible (BOGDEN & TAYLOR, 1975; MAYKUT & MOREHOUSE, 1994). [17]

CROTTY (1996) suggests that all qualitative research is unique. This we acknowledge. While the underlying principles of our phenomenological approach did not change, we are prepared to accept that we may not have remained entirely true to the hermeneutic principles of phenomenological research throughout the project. The research therefore might be considered by some to be more accurately labelled naturalistic rather than phenomenological. [18]

It has been argued that the conventional criteria for judging the rigour or trustworthiness of qualitative research which include internal validity, external validity, reliability and objectivity are not always appropriate. GUBA and LINCOLN (1989) and LINCOLN and GUBA (1985) described these facets as follows:

* Internal validity: the extent to which variations in an outcome or dependent variable can be attributed to controlled variation in an independent variable

* External validity: inference that the presumed causal relationship can be generalised across alternate measures of cause and effect and across different types of persons, settings and times

* Reliability: consistency of a given inquiry--is generally a precondition for validity. It refers to a study's consistency, predictibility, dependability, stability and/or accuracy. Reliability typically rests on replication

* Objectivity: neutrality, a demonstration that the inquiry is free of bias, values and/or prejudice [19]

We have used GUBA and LINCOLN's (1989) alternative criteria to assess the rigour of our research: [20]

Credibility

GUBA and LINCOLN (1989) describe credibility as being parallel to internal validity. The focus is more on establishing the match between the constructed realities of respondents and those realities as represented by the evaluator and attributed to various stakeholders. Credibility can be "verified" by 1) prolonged engagement; 2) persistent observation; 3) peer debriefing; 4) negative case analysis; 5) progressive subjectivity; 6) member checks. We consider our research to be credible through a number of mechanisms. [21]

The researchers, in the main, knew the pastoralists that participated in the study which allowed for a measure of trust between the researchers and those being researched. This helps establish the match between respondents and evaluators' views of reality as well as a prolonged engagement and persistent observation. The research team included four "field" researchers and a further two full time researchers with expertise in adult education and learning processes. This provided an opportunity for "disinterested" peer reviews of the interviews and produced extensive discussion and analysis of the data throughout the research project. This equates to verification by peer debriefing as suggested above. Four other researchers had peripheral involvement in the study and provided feedback to the research team at various stages of the project and this strengthened the peer debriefing check and introduced some reviewed subjectivity into the study. An additional aid to ensuring rigour through credibility was to include the pastoralists in later stages of the study, by sending the transcripts of the interviews for verification and then providing them with drafts of the research outcomes. A final stage will be for the field researchers to make presentations of the research findings to various pastoral groups. These member checks are the strength of this type of participative research project. [22]

Transferability

Transferability is described as being parallel to external validity or generalisability. Transferability is relative and depends entirely on the degree to which salient conditions overlap or match. This is mostly verified through "thick" description. The constructivist does not provide the confidence limits of the study, but instead provides as complete a database as possible in order to facilitate transferability judgements on the part of others. This measure of rigour has been enacted through making this a multi-site project. [23]

Although one of the intentions at the beginning of the study was to undertake an analysis of differences between the three States, the results have shown very minimal differences. The conditions for learning by the pastoralist families for the most part overlap and the rich descriptions that can be produced from this study will, in all likelihood, be applicable to most families in, at least, the tropical savannas of northern Australia. Most of these differences came from the sources of information and that would be expected given the wide geographical spread of the families. However, even these did not differ very much. For example, pastoral families from all three States stated they used sources of information from Queensland (although Queensland pastoralists did not go outside their State to seek further information). [24]

Dependability

Dependability is parallel to reliability and likewise concerned with the stability of the data over time. Researchers need to be able to demonstrate any changes or shifts in the way in which the inquiry was conducted. This measure was ensured through maintaining constant contact between members of the research team at all stages of the study, with three workshops (although the third one consisted of only four of the core team of six) and around 15 tele-conferences as well as contact through e-mail (sending of documents). The tele-conferences, in particular, were used to ensure that the research was being conducted in a similar manner across the three States, and then that the data was being consistently analysed. It was also important for all the researchers to analyse all the data, not just that they had collected themselves. [25]

Confirmability

Confirmability is described as being parallel to objectivity. It is the need to show that data, interpretations and outcome of inquiries are rooted in contexts and persons apart from the evaluator and are not simply figments of the evaluator's imagination. All data needs to be able to be tracked to its source and that the logic used to assemble the interpretations into structurally coherent and corroborating wholes is both explicit and implicit in the narrative of the case study. [26]

While accepting that there is an inevitable element of subjectivity in constructivist research, in some instances researcher comments could be construed as leading, offering opinions or explanations that the pastoralists happily took on as their own. However, these instances were moderated and checked through having a research team of six. The discrepancies were recognised and removed. As much of data analysis also took place as a workshop, any individual biases were mediated. The fact that this was a multi-site, multi-discipline and multi-researcher study ensured rigour in this context. This workshop process (the subject of a future paper) ensured that the interpretations of the data were contained within the context of the pastoral family's explanation and were not, as GUBA and LINCOLN (1989) put it figments of the evaluator's imagination, or in our case constructed by the evaluator for the benefit of the pastoral family. [27]

4. Conclusions

Measuring our research against GUBA and LINCOLN's 1989 criteria suggest that we have been able to maintain rigour within our research despite the difficulty of the different sites, researchers and backgrounds. Continual communication, a simple adherence to the principles of our research methodology, and listening to our fellow farmer researchers has strengthened this. We would also like to postulate that the straightforward description of what we have undertaken offers and guarantees a degree of rigour for our research. [28]

In short, we have described to you our research process and through that description allowed the research to be repeated. That is all we can do, as the final understanding of the learning process is ours. It is up to you the reader to agree with the interpretation, and when reading the full report to decide if that is as you see it or not. As described in this paper, our rigour is maintained in the structure of our methodology and in the full description of what and how we did the research. [29]

Authors

Dr. Ken CRAWFORD is a Development Officer and Project Manager with Agricultural Western Australia, the State Government Agency dealing with agricultural and sustainable resource management issues. He has post-graduate qualifications in Farm management from Massey University, Palmerston North New Zealand and is undertaking PhD research into "How farmers learn" in the Western Australian broadacre production environment. He has undertaken research on agricultural issues in New Zealand and Australia.

Contact:

Ken Crawford

E-mail: [email protected]

Dr. Marnie LEYBOURNE is a Senior Policy Officer with Water & Rivers Commission, a State Government Agency in Western Australia and deals with natural resource management issues. She has post-graduate qualifications in Development Studies from University Institute of Development Studies in Geneva and a PhD in Geography from Lyon University, France. She has undertaken research on agricultural and pastoral societies in Syria, Switzerland, New Zealand and Australia.

Contact:

Marnie Leybourne

E-mail: [email protected]

Dr. Allan ARNOTT is currently the Deputy Director for the Centre for Teaching and Learning in Diverse Educational Contexts, and works University. He spent several years in remote areas of the Northern Territory of Australia establishing and operating adult education services in both Aboriginal settings and in mining towns. Present research interests include professional development needs of remote area adult educators, modes of educational delivery in remote and rural Australia, and workplace learning.

Contact:

Allan Arnott

E-mail: [email protected]

Citation

Crawford, H. Ken; Leybourne, Marnie L. & Arnott, Allan (2000). How we Ensured Rigour in a Multi-site, Multi-discipline, Multi-researcher Study [29 paragraphs]. Forum Qualitative Sozialforschung / Forum: Qualitative Social Research, 1(1), Art. 12, http://nbn-resolving.de/urn:nbn:de:0114-fqs0001125.

Revised 3/2007

© 1999-2011 Forum Qualitative Sozialforschung / Forum: Qualitative Social Research (ISSN 1438-5627)

Supported by the Institute for Qualitative Research and the Center for Digital Systems, Freie Universität Berlin