Approaches to address the lack of relevance in technology acceptance research for mobile services.

Headnote

Abstract. This paper aims to analyze the state-ofthe- art of technology acceptance research in the area of mobile services concerning rigor and relevance as well as concerning the scientific approach that is used based on a full text analysis. The results show the current dominance of rigor over relevance and of Behavioral Science over Design Science but also offer opportunities to enhance research quality as a whole by including Design Science principles in this research area that is usually seen as a typical domain for Behavioral Science.

Keywords. Technology acceptance research, mobile services, rigor versus relevance, full text analysis

1 Introduction

Technology acceptance research is commonly allocated in the area of Behavioral Science [13] as it focuses the user's behavioral intention to use a certain technology. Various models of technology acceptance were published in the course of the last decades. Most popular among them is Technology Acceptance Model [8] but also Task Technology Fit Model [11], Unified Theory of Acceptance and Use of Technology [21] and still Theory of Reasoned Action [10] are quite often used. An increasing number of papers in this area addresses acceptance of mobile technologies and services. Unfortunately there seems to be a lack of relevance as many innovations fail. As to say practitioners are unable to predict user acceptance of mobile services. Examples are Mobile TV via dvb-h in Europe which was predicted to be a "killer application" or SMS which was supposed to be a niche product. Either the models don't work in these cases or they don't even reach the people they address. A pilot study [16] examining two volumes of important IS journals already confirmed the assumption that there is a lack of relevance in acceptance research for mobile services.

This problem led to following research questions:

* Is rigor still dominating relevance?

* Are there any attempts to include the Design Science approach in technology acceptance research?

* Are rigor and relevance contradictorily qualities?

* Is rigor the monopolized by Behavioral Science and relevance by Design Science?

* Are Design Science and Behavioral Science diametrically opposed approaches?

* Are there possibilities to include both and thereby enhance quality of research?

2 Conceptual Framework

Most ideas of this study are drawn from two important debates that influenced the IS community. One is the Rigor versus Relevance-debate that peaked in the late 1990s and second is the ongoing Behavioral versus Design Science-debate that includes arguments deriving from the first.

2.1 Rigor vs. Relevance

Rigor is a quality criterion concerning scientific standards, norms and commons. It is influenced by many different factors and is delimited in various ways. Some parameters of rigor occur quite often in papers dealing with rigor and relevance and they seem to be essential. Replicability of research [19; 4; 27] is one of them. The research process should be designed and described in a way that enables other researchers to repeat it. Logical rigor [28; 23; 24; 1; 4; 18; 7; 15; 22] is also a commonly named factor. Modeling and derivation of hypothesis and research model should be logically traceable. Coverage of significant literature [13; 22; 19; 4; 27; 18] is a basic criterion for rigorous research. It implies the requirement to include existing theories and thoughts and not to neglect important literature. The call for reasonable reliability and validity tests [20; 1] includes an argumentative base for their use. Sample choice [19; 6] is a disputed topic in the area of rigor. There exist different attitudes concerning so called "magical numbers" that indicate whether "n" is sufficiently high or not. Also the composition of the sample can influence research quality.

Relevance as a second important quality criterion pertains to practical usefulness of research. A high relevant paper will influence people's behaviour, e.g. concerning design and implementation of a new technology, in practice. Insufficient rigor can also harm relevance as research that is not done properly is hardly applicable [25]. Other factors are not so easy to delineate. A relevant paper should offer precise implications for action [12; 28; 24]. The reader should find clear instructions of processes and activities that are suggested. Timeliness of the underlying technology [13; 12; 14; 24] is also important and often named. Research can be more influential for new or upcoming technologies than for old and already outdated ones. The success factors [22] that are proposed should be counterintuitive or at least debateable. Otherwise no practitioner will consider the paper relevant. Limitations [22] of the study have to be stated explicitly to define applicability of the results. A further point that is crucial to relevance is addressing the target group [13; 2]. Research that is not communicated properly to its addressees is not relevant for them. Therefore language, content and journal choice should be based on the intended target group.

2.2 Behavioral Science vs. Design Science

Research intention is the main difference between Behavioral and Design Science approaches [14]. Design Science on the one hand aims to contribute to practice [14; 28; 19; 4; 27; 18; 3], whereas Behavioral Science on the other hand wants to contribute to knowledge [13; 14; 28; 19; 4; 27; 18; 22]. The first intends to solve problems rather than understand them [1;14]. Also the sources of problems are different. While Design Science draws problems from real world and practice, Behavioral Science addresses scientific problems [13]. Main activities in Design Science are creation and evaluation [14] whereas a Behavioral Science approach will require to discover and to justify [13; 14]. Generalizability is a goal of Behavioral Science research [13; 14] but not of Design Science where results are mainly limited to one specific case. Design Science necessitates an iterative research process [13; 25] where the artefact is evaluated and reevaluated while Behavioral Science papers are often based on one finished study. Behavioral Science research on the one hand tests validity concerning content, constructs and items [5; 20; 14]. Design Science on the other hand seeks to test validity concerning reality [13; 3]. As to say test if the artefact is useful in order to solve the problem.

2.3 Research Model

Based on the assumptions mentioned above and on the suggestion to differentiate between research process and research product [9] the research model depicted in figure 1 was elaborated.

Design Science and Behavioral Science are regarded as to approaches that can be followed in the research process. Rigor and relevance are assumed to be two qualities of the research output. Of course rigor and relevance are important qualities of the research process too but as the problem discussed in this paper is predominantly a problem of communicating relevant research both are only regarded in the capacity of article qualities.

2.4 Hypotheses

According to the research model and research questions above interrelations concerning different topics were hypothesized.

Concerning distribution of occurrence:

1a. Rigor exceeds relevance.

1b. Behavioral Science approach exceeds Design Science approach.

Concerning dichotomy of rigor and relevance:

2a. The more rigor, the less relevance.

2b. The more relevance, the less rigor.

Concerning dichotomy of Behavioral Science and Design Science:

3a. The more Behavioral Science-critera are met, the fewer Design Science-critera are met.

3b. The more Design Science-critera are met, the fewer Behavioral Science-critera are met.

Concerning relatedness of Rigor vs. Relevance problem and Behavioral Science vs. Design Science problem:

4a. The more Behavioral Science criteria are met, the more rigorous is the paper.

4b. The more Design Science-critera are met, the more relevant is the paper.

3 Study Design

The study is based on a full text analysis of articles concerning technology acceptance of mobile services published in main IS journals since 2004.

3.1 Journal Selection

The journals considered for this research were selected on the base of three criteria. First criterion was the "Science Citation Index"-ranking. All Journals from category "Information Science & Library Science" in SSCI or "Computer Science, Information Systems" in SCI with ISI-JCR-Impact factors higher than 1.000 were included. Also Journals with A-rankings by the "Wissenschaftliche Kommission Wirtschaftsinformatik im Verband der Hochschullehrer für Betriebswirtschaft" (= part of the German Academic Association for Business Research) and "Fachbereich für Wirtschaftsinformatik der Gesellschaftfür Informatik" (= part of the German Society for Informatics) published in "Wirtschaftsinformatik" [26] were chosen. Based on the third criterion also journals ranked as "BJournals" in the list mentioned above and those with impact factors lower than 1.000 but included at least one significant article in the pilot research were included for further examination.

3.2 Article Selection

All articles concerning technology acceptance research for mobile services were chosen. For this purpose all tables of content and if necessary abstracts were screened for articles that deal with adoption and further usage of mobile services. Mobile services were regarded in a broad sense as to say all applications and functions on mobile devices were included.

3.3 Encoding process

All selected articles were encoded according to the criteria catalogue listed in table 1 and table 2 below. Each item was rated on a scale from 0 - poorly met criterion to 5 - highly met criterion. The encoding process was administered independently by two researchers that exchanged their experiences after encoding a randomly chosen sample of 15 articles. Their results were fairly close and the discussion led to further accordance.

4 Results and Discussion

The 67 journals considered for this study published a total amount of 25031 articles in the period from 2004 to 2009. 73 of these articles dealt with the topic in question which represents 0,29 % of all articles. Figure 2 below depicts the chronological distribution of papers.

4.1 Rigor vs. Relevance

On average papers received 1,83 points on the relevance scale and 2,35 points on the rigor scale. In total numbers 39 papers were predominantly rigorous whereas only 28 papers were predominantly relevant.

These result supports hypothesis 1a: Rigor exceeds relevance. Mean values of all single items are listed in table 3 below. Coverage of significant literature was the quality criterion with the highest mean value and timeliness with the lowest value. This might indicate that sufficient rigor is a conditio sine qua non whereas relevance is an additional quality.

In figure 3 below all papers are positioned in a graph that depicts their mean values concerning rigor and relevance. If rigor and relevance were completely contradictorily qualities all papers should be located on the two axes of the graph. This is not the case. Many papers are located between the axes and therefore include both qualities. The correlation of rigor and relevance is postive (0,56) not negative as suggested in hypotheses 2a and 2b.

4.2 Behavioral Science vs. Design Science

Altogether 44 papers were dominated by a Behavioral Science approach while only 19 papers were basically Design Science papers. Mean value for Design Science characteristics was 0,78 and 1,82 for Behavioral Science. This supports hypothesis 1b: Behavioral Science approach exceeds Design Science approach. Results for all single items are listed in table 4 below. The intention to contribute to knowledge as well as the activity to discover or justify received highest mean values in this concern whereas the desire to solve problems from real world received the lowest mean value in this concern.

Unlike rigor and relevance there seems to be a dichotomy concerning Design Science and Behavioral Sciences. All papers are again positioned in a graph depicting their Behavioral Science and Design Science properties. 43 out of 44 Behavioral Science papers do not meet any Design Science criteria and 14 out of 19 Design Science papers do not include any Behavioral Science principles. There is also a negative correlation (-0,46) between the two approaches. These results support hypotheses 3a and 3b concerning the dichotomy of Design Science and Behavioral Science. Nevertheless some papers included characteristics and principles of both approaches.

Concerning the relatedness of the two problems in concern namely rigor versus relevance and Design Science versus Behavioral Science results are not too clear. When regarding the 44 Behavioral Science Papers there is a significant dominance of rigor observable. 33 of them are predominately rigorous papers and the mean value for rigor is 3,17 (compared to 2,35 for all papers). There is also a strong correlation (0,85) between rigor and Behavioral Science characteristics. These results support hypothesis 4a: The more Behavioral Science criteria are met, the more rigor.

In the case of Design Science and relevance the relation is not that simple. Out of 19 Design Science papers 15 were predominantly relevant which indicates a relation between these properties but the correlation between Design Science characteristics and relevance is not significant. Nevertheless there seems to be at least some kind of relatedness in this case as well.

5 Summary and Outlook

The problem addressed by this paper is missing relevance of technology acceptance research for mobile services. In order to enhance relevance the possibilities concerning inclusion of Design Science principles is examined. Table 5 below gives an overview of the results of the study.

The results show that there is a lack of relevance in technology acceptance research and that there is no compulsory contradiction of rigor and relevance. Also possibilities for the coexistence of Behavioral Science and Design Science were detected which might hold opportunities for research in future. Relevance must be increased in order to foster research quality. One path to achieve this goal is the inclusion of Design Science principles into the research process. This leads to new requirements for technology acceptance research for mobile services:

* Research aims to contribute to practice as well as to common knowledge.

* Research aims to solve problems from real world as well as understand scientific problems.

* Usefulness is a validity criterion concerning reality next to content, construct and item validity.

* Success factors are new and discussible.

* Research concerns new or upcoming technology.

* Limitations of the results are clearly formulated.

* The paper aims to and is designed to reach its defined target group.

* The paper offers precise implications for action.

All these new criteria should be met while sticking to existing "old" criteria of rigor and Behavioral Science (e.g. coverage of significant literature, reasonable reliability and validity tests, logical rigor etc.). This approach could enhance quality of research and prevent expensive innovation failures.

The research gap detected in the course of this study will bet the foundation for further work in the area of technology acceptance research for mobile services. A technological artefact that measures factors influencing user acceptance in real-time should foster relevance in this field. Relevance will be high lightened by including Design Science principles while preserving rigor on a reasonable level.

Simultaneously to this study data concerning basic research models, used acceptance constructs and main results of the publications in the area was collected. Analysis of this data will give an overview of the state-of-the-art in the area [17].

Further research could emphasize other aspects of Design Science, Behavioral Science, rigor and relevance as the ones used in this study are not exhaustive. A research question in this area could be: Which criteria of Design Science and Behavioral Science are contradictorily and cannot coexist in one research process? As this study showed both approaches can coexist in general but there might be some criteria that are completely opposite. The same question could be interesting for rigor and relevance.

6 Acknowledgments

Special thanks from all authors to Anja Tuppinger who operated the abstract screening and supported the encoding process.

References

References

[1] Bacharach, S.B.: Organizational Theories: Some Criteria for Evaluation, Academy of Management Review 14, 1989, pp. 496-515.

[2] Benbasat, I., Zmud, R.W.: Empirical Research in Information Systems: The Practice of Relevance, MIS Quarterly 23, 1999, pp. 3-16.

[3] Bennis, W.G., O'Toole, J.: How Business Schools Lost Their Way, Harvard Business Review 83, 2005, pp. 96-104.

[4] Chase, J.M.: Normative Criteria for Scientific Publication, The American Sociologist, August, 1970, pp. 262-265.

[5] Chin, W.W., Todd, P.A.: On the Use, Usefulness, and Ease of Use of Structural Equation Modeling in MIS Research: A Note of Caution, MIS Quarterly 19, 1995, pp. 237- 246.

[6] Cohen, J.: Statistical Power Analysis for the Behavioral Sciences, Lawrence Erlbaum Associates, Hillsdale, NJ, 1988.

[7] Daft, R.L.: Why I Recommended That Your Manuscript Be Rejected and What You Can Do about It, In: Cummings, L.L., Frost, P.J. (eds.) Publishing in the Organizational Sciences. Irwin, Homewood, IL, 1985, pp. 193-209.

[8] Davis, F.: A Technology Acceptance Model for Empirically Testing New End-User Information systems, Massachusetts Institute of Technology, Sloan School of Management Thesis, 1985.

[9] Fällman, D., Grönlund, Å.: Rigor and Relevance Remodeled, In: Proceedings of Information Systems Research in Scandinavia, IRIS25, Bautahoj, Denmark, August 10-13 2002.

[10] Fishbein, M., Ajzen, I.: Belief, Attitude, Intention and Behavior: An introduction to Theory and Research, Reading, Addison- Wesely, 1975.

[11] Goodhue, D.L., Thompson, R.L.: Task- Technology fit and individual performance, MIS Quarterly 19, 1995, pp. 213-236.

[12] Gregor, S.: The Nature of Theory in Information systems, MIS Quarterly 30, 2006, pp. 611-642.

[13] Hevner, A.R, March, S.T., Park, J.: Design Science in Information Systems Research, MIS Quarterly 28, 2004, pp. 75-105.

[14] March, S.T., Smith, G.F.: Design and natural science research on information technology, Decision Support Systems 15, 1995, pp. 251-266.

[15] Mitchell, T.R., Beach, L. R., Smith, K.G.: Some Data on Publishing from the Authors' and Reviewers' Perspectives, In: Cummings, L.L., Frost, P.J. (eds.) Publishing in the Organizational Sciences. Irwin, Homewood, IL, 1985, pp. 248- 264.

[16] Platzer, E.: A critical review of user acceptance research in the area of mobile services, Libri 59, 2009, pp. 213-227.

[17] Platzer, E., Petrovic, O.: Development of technology acceptance research for mobile services, In: Proceedings of Mipro 2010, Opatija, Croatia, May 24-28 2010 (accepted paper).

[18] Price, R.L.: A Customer's View of Organizational Literature, In: Cummings, L.L., Frost, P.J. (eds.) Publishing in the Organizational Sciences. Irwin, Homewood, IL, 1985, pp. 125- 133.

[19] Straub, D.W.: Editor's Comments: Why Top Journals Accept Your Paper, MIS Quarterly 33, 2009, pp. iii-x.

[20] Straub, D.W.: Validating Instruments in MIS Research, MIS Quarterly 13, 1989, pp. 147-169.

[21] Venkatesh, V.: User Acceptance of Information Technology: Toward a Unified View, MIS Quarterly 27, 2003, pp. 425-478.

[22] Weber, R.: Editor's Comments: Theoretically Speaking, MIS Quarterly 27, 2003, pp. iii-xii.

[23] Weick, K.E.: What Theory Is NOT, Theorizing IS, Administrative Science Quarterly 40, 1995, pp. 385-390.

[24] Whetten, D.A.: What Constitutes a Theoretical Contribution?, Academy of Management Review 14, 1989, pp. 490-495.

[25] Winter, R.: Interview mit Alan R. Hevner zum Thema "Design Science" (Interview with Alan R. Hevner about "Design Science"). Wirtschaftsinformatik 51, 2009, pp. 148-152.

[26] WKWI, GI-FB WI: WI-Orientierungslisten. WI-Journalliste 2008 sowie WI-Liste der Konferenzen, Proceedings und Lecture Notes 2008 (WI-lists for orientation. WI-journal list and WI-list of conferences, proceedings and lecture notes). Wirtschaftsinformatik 50, 2008, pp. 155- 163.

[27] Wolff, W.M.: A Study of Criteria for Journal Manuscripts, American Psychologist 25, 1970, pp. 636-639.

[28] Zmud, B.: Editor's Comments: "Pure" Theory Manuscripts, MIS Quarterly 22, 1998, pp. xxixxxxii.