1. Introduction
The rapid evolution of technology has brought new opportunities for developing new applications and for interacting with them. Augmented Reality (AR) has become an alternative way to engage users with applications more efficiently [1]. Kipper and Rampolla [2] define Augmented Reality (AR) as a blending of real-world and digital information. This combination of the real world and the virtual world has been harnessed for various purposes, such as marketing, education, entertainment and health care. For example, AR has been used for educational and training purposes across subject areas both in formal and informal settings [3,4,5,6,7,8], and it has been shown to possess a multitude of affordances for learning and training scenarios [7,8,9,10,11]. Despite these success stories, AR (and mobile AR in particular) has not yet fulfilled a previously made prediction of becoming the eighth mass media [12]. There is a significant body of research on AR capabilities in different sectors [13], and the challenges and limitations of AR have been discussed [7,14].
Aligned with the popularity of smart phones and tablets, Mobile Augmented Reality (MAR) has emerged as a mainstream technology and is currently a hot topic among researchers across fields. As in our earlier work [3], MAR is defined here as “a type of AR where a mobile device (smartphone or tablet) is used to display and interact with virtual content that are overlaid on top of a real-time camera feed of the real world”. Virtual content in MAR applications can be context-sensitive and take various forms including, but not limited to three-dimensional (3D) models, animations, annotations and videos. Recently, popular mobile applications with AR features, such as Pokémon Go and Snapchat, have changed the developers’ and users’ perceptions of AR. These applications brought MAR to people’s smartphones in earnest, especially targeting kids and youth. Moreover, the increase in the number of toolkits for creating MAR applications has enabled even non-technical individuals to create their own MAR content [3], thus further strengthening the position of MAR as a mainstream interaction technology.
Although Pokémon Go has been associated with many benefits, such as making people exercise more [15] and helping those who have withdrawn from social life [16], little attention has been put on investigation of User Experience (UX) factors that contribute to positive reception of MAR applications. Following our earlier work [17], UX is defined here as the emotions that the user encounters while using a service, a product or an application. An example of this is the negative emotions experienced by some Pokémon Go players when the AR feature of the game drains the smartphone battery and makes their smartphones slow [18,19].
Although UX of MAR applications have received some attention from researchers (e.g., [20,21]), there is a need for a study that focuses on MAR UX from the perspective of emotions. To address this need, the main objectives of this study are to assess the UX of two MAR applications and to draw conclusions as to what challenges, opportunities and best practices are associated with MAR application development from the UX perspective.
The rest of the paper is organized as follows. We start with an overview of previous research on mobile user experience and its relation to mobile augmented reality. Then, after presenting the research design, we describe two MAR applications—a commercial AR experience and our own Virtual Campus Tour application—that are used as case studies in this paper. In the Results Section, we present the results of mixed-method evaluations of the two case studies and then describe challenges, opportunities and best practices associated with mobile augmented reality user experience. The identified best practices are then utilized to improve the second case study. Before concluding the study, we provide discussion on the findings.
2. Background
2.1. Defining Mobile User Experience
Despite a more than a decade’s worth of research on UX, the concept still suffers from vague and broad definitions, which is reflected in the 27 definitions of UX reported as the result of the Dagstuhl Seminar of Demarcating User Experience [22]. UX is a multidimensional phenomenon [23] in which many factors influence success, and it has been around for many years [17]. Carlos et al. [24] identify users, context, interaction and artifacts as the key elements of UX. Moreover, Hassenzahl [25] refers to UX as the “quality of interactive technology”, focusing on the human and not on the product. Hassenzahl and Tractinsky [26] believe that with the advancement of technology, interactive products and services become not only useful and usable, but also trendy and fashionable. In this study, UX is defined as the emotions that the user encounters while using a service, a product or an application [17]. This definition is a consequence of the recent development of design thinking, which recognizes emotional engagement as a key factor in enabling sustainable use of mobile applications.
In mobile applications, UX requires special attention due to limitations of mobile devices (e.g., screen size) and the context-awareness afforded by the embedded sensors. Various attempts have been made to promote mobile UX through evaluation metrics and design approaches in recent years. Yong [27] recommends UX evaluation methods for mobile devices and proposes that these methods should measure the emotional responses of users on visceral, behavioral and reflective dimensions. Väätäjä and Roto [28] recommend mobile questionnaires for mobile UX evaluation, but point out that in the design of the questionnaires, one must pay special attention to the clarity and simplicity of the structure including questions, icons, items and scales. Pelet and Taieb [29] focus on mobile application design to promote UX, with recommendations on how color contrast can be used to impact the positive effects on behavioral intentions. Furthermore, Dirin and Nieminen [30] propose a usability and UX framework for mobile learning applications. The framework has been proven to promote UX in a mobile-based application by making the application adjustable, delightful and reliable, all of which are factors contributing to the emotional dimension of UX.
The existing mobile UX measurement and design approaches may also be to some extent applied to MAR applications. However, MAR UX design is not the same as mobile UX design. Although emotional engagement, personalization and reliability are important in all applications and services, in MAR UX, users are engaged both mentally and physically in the application. The mental engagement in this context means that users fully concentrate on the application, and short- and long-term memory are engaged in the application. Physical engagement means that users often need both hands, eyes, ears and even the rest of their bodies to be involved with the application.
2.2. Mobile Augmented Reality and User Experience
Previous studies on MAR from the UX perspective are vague, and there is certainly room for further research [31]. The main reasons for the lack of MAR UX research and design guidelines is the recentness of MAR as a mainstream technology. Major advancements in the development of MAR happened during the last decade [32]. The main breakthrough occurred with the emergence of smartphones equipped with advanced technologies [33]. Yet, there are some researchers who have investigated the area of MAR UX, as we briefly explain in the following.
Irshad et al. [31] conducted a review on MAR UX research published in 2005–2014 and presented a categorization of the MAR UX domain, including aspects such as UX as a phenomenon, UX as a field of study and UX as a practice. Furthermore, they promoted User-Centered Design (UCD) as a recommended method for MAR application development. When we reviewed previous research on MAR UX, we discovered that MAR applications, where UX was specifically considered, were developed and researched either through case study evaluations or through MAR application development frameworks. In both cases, the user of the application is typically at the center of the design and development, thus confirming the results of Irshad et al. For example, Rashidi et al. [34] proposed a framework based on UCD for developing MAR tourism applications. In another example, Olsson et al. [21] and Olsson and Väänänen-Vainio-Mattila [35] elaborated through a case study MAR application and came to the conclusion that MAR applications’ acceptance by potential users requires that designers learn about users and their essential needs in advance. This is not unlike most UCD frameworks that involve end-users early in the design process.
Furthermore, usability (i.e., effectiveness, efficiency, satisfaction), which is an important factor in positive UX, has traditionally been associated with mobile application design and development. Aligned with this, Ritsos et al. [20] advocated the importance of usability as a key factor for the adoption of MAR applications by users, whereas Jung et al. [36] focused on the satisfaction attribute of usability while conducting a study on a marker-based MAR application for tourists. The findings of Jung et al. indicated that content, personalized service and system quality affect users’ satisfaction.
To promote MAR usability and UX, some researchers have focused on framework development. For example, Kim [37] proposed a framework for context immersion in MAR applications. The framework is constructed of three contexts: time- and location-based context, object-based context and user-based context. Understanding users’ needs in these contexts is a crucial part of UX design in MAR applications. Aligned with this, Irshad and Awang Rambli [38] proposed a multi-layered MAR framework for achieving positive UX. Their framework is divided into various dimensions of a MAR application, such as product, time, specific context and UX.
Our literature review revealed that MAR UX is attracting interest among researchers, with most of the related publications having been published in recent years. Previous studies indicated that the focus has been on UX and usability evaluation of MAR applications, but we still lack information on how users perceive MAR applications emotionally and what are the best practices for MAR UX designers. Important questions to answer include: “What kinds of emotions arise among users of MAR applications?” and “What concerns do users have when they deal with MAR applications?” The aim of this paper is to answer these questions through evaluations of two MAR case studies.
3. Research Design
The following sections describe the research questions, participants and methods used in the evaluation of the two case study applications, which are described in Section 4.
3.1. Research Questions
It is evident that UX of a mobile application is different from that of a MAR application [39]. To investigate the role of UX in the development and evaluation of MAR applications, this study aims to answer the following research questions through two case studies (see Section 4):
1. How do users perceive two MAR case study applications emotionally?
2. What are the major opportunities and challenges associated with MAR UX?
3. What are the best practices for MAR UX?
We chose to analyze the case studies by a mixed-method approach comprising qualitative and quantitative methods. We explain these methods in the following sections after describing the study participants.
3.2. Participants
The case study evaluations were conducted at the Haaga-Helia University of Applied Sciences in March 2018. Table 1 summarizes the demographics of the participants for both case studies. The participants were recruited on a voluntary basis from a user experience design course at Haaga-Helia. No rewards were given to the participants. All collected data were anonymized so that the participants cannot be identified from the results presented in this study.
[ Table omitted. See PDF. ]
The participants were experts in using information technology, thus no technology training was provided to them before the experiments. Many participants used Android smartphones, and a few owned iPhones. The experiments were conducted in English because the participants were from different countries.
It is noteworthy that although the sample sizes are relatively small, and we therefore cannot draw statistically-significant conclusions, the quantitative and qualitative results together indicate useful information on the UX of the two case studies. The number of recommended test users for usability experiments varies between three and five. For example, Nielsen [40] states that with five test users, one almost gets close to the maximum benefit-cost ratio in usability testing. Nielsen [40] further promotes qualitative user research for collecting insights to drive the design, not the number of involved people.
3.3. Research Methods: Case Study 1
3.3.1. Data Collection
In Case Study 1, we asked the participants to experiment with a commercial MAR application featuring an interactive cat (described in Section 4.1) and then administered a questionnaire based on [41] about emotions that the participants felt during the experiment. The questionnaire consisted of basic information, such as gender, age and previous experiences with MAR. Additionally, the questionnaire had Likert-scale questions about the participant’s emotions during and after the experiment. Figure 1 presents samples of the Likert statements used for collecting data on the participants’ emotions.
Enlarge this image.Figure 1. Samples of Likert-scale statements in the questionnaire of Case Study 1.
We also conducted short interviews with selected participants about their experiences with the application. We randomly selected nine participants (n = 9, females: 5, males: 4) for interviews among those who finished the experiment in 3–5 min (n = 4), in 6–10 min (n = 3) and in more than 10 min (n = 2). The interviews were recorded as audio and transcribed later. The following questions were asked in the interview:
1. Question #1: What was your first impression when you saw the cat?
2. Question #2: Did you find it difficult to test the application and bring the cat to the screen?
3. Question #3: Would you like to have this type of application in your daily life?
4. Question #4: How would you change this application?
The experiment was conducted in a classroom with the following procedure. The participants were given smartphones (Samsung Galaxy S6) and were instructed to download the Arilyn application from Google Play. The participants were then asked to scan the cat picture on a milk package (we had two empty packages of milk in the classroom) or through a cat picture, which, for the sake of accessibility, was presented to the participants through a PowerPoint slide. The duration of the experiment varied between 3 and 15 min among the participants. The participants were given freedom to explore the functionality of the MAR application; therefore, no specific instructions or tasks were given. The researchers observed the participants during the experiment and took notes accordingly. After the participants finished the experiment, they were asked to fill in a questionnaire that was delivered as a hardcopy. After the experiment, a researcher conducted interviews with the selected participants.
3.3.2. Data Analysis
In Case Study 1, we applied statistical tools to the collected quantitative data to assess how the participants felt about the MAR application. The collected questionnaire data on emotions was based on a Likert-scale; thus, a simple proportional analysis of the participants’ answers was considered to be an appropriate approach to represent the users’ emotional states towards the tested MAR application. To analyze the questionnaire data, we used the ToolPak add-in for Microsoft Excel, which provides various data analysis tools for statistical and engineering analysis.
After transcribing the interview data, the participants’ answers and researchers’ observations were analyzed to learn more about the participants’ perceptions and reactions towards the MAR application. The analysis was conducted by applying the method of coding [42] in a word processor whereby keywords were assigned to the qualitative data excerpts. These keywords were, for example, emotional words that the participants mentioned or the emotional expressions that were extracted by observing. The frequencies of the keywords are not reported in this study due to the relatively small sample size.
3.4. Research Methods: Case Study 2
3.4.1. Data Collection
In Case Study 2, the participants used a Virtual Campus Tour application that was designed and developed at the Haaga-Helia University of Applied Sciences (see more details in Section 4.2). We conducted a usability and UX evaluation in a usability lab at the Haaga-Helia University of Applied Sciences. The usability evaluation focused on the performance of a set of predefined tasks to measure the efficiency of the application. The UX evaluation, which was based on observations and post-experiment interviews of the participants, focused on how the participants felt about the MAR application. The aim of this case study was to investigate the emotions that the participants felt about the application and to find out whether the participants found the application easy and efficient to use and whether it fulfilled their essential needs. Table 2 presents the details of the test plan.
[ Table omitted. See PDF. ]
In the usability assessment, which was video-recorded, we applied the think-aloud method to learn about the participants’ ways of performing the given tasks. In the think-aloud method, which is rooted in psychology [43], the participants are advised to say what they think while performing a given task. In our experiment, the objective of using the think-aloud method was to give us feedback on whether the interface is logically designed and whether the participants find the components in expected places.
There were two people conducting the Case Study 2 experiment: the facilitator and the observer. The facilitator briefly introduced the application prototype and the purpose of the test to the participants and answered any questions that the participants had about the experiment. During the experiment, the facilitator assisted the participants if they had problems, but without guiding them to solve the tasks independently. Finally, the facilitator interviewed the participants. The observer silently recorded the times of the tasks and observed the verbal language and facial expressions of the participants. The observer also took notes on the participants’ actions, comments, procedural errors and problems and assisted the facilitator in writing down the participants’ interview answers.
3.4.2. Data Analysis
To analyze the data collected in Case Study 2, we applied usability evaluation metrics, such as the time spent on performing the predefined tasks and the number of tasks performed in a given time, to measure the application’s efficiency. Furthermore, the qualitative dataset comprised transcribed think-aloud videos, observations and interview recordings. As in Case Study 1, we applied coding [42] in a word processor to identify emotional reactions and usability issues raised by the participants during the experiment. The frequencies of the keywords are not reported in this study due to the relatively small sample size.
4. Case Studies
4.1. Case Study 1: Aamu Cat
The first case study is a commercial advertisement MAR application by a Scandinavian dairy corporation (Arla), which was in the market during a promotional period (in 2018) in Finland. The customers were advised to download the application (Arilyn) to their phones and scan a picture of a cat embedded in a milk package, thereby activating the AR content. The AR content features a cat, called Aamu (morning in Finnish), that jumps to the phone screen out of the milk package, thus granting the user the opportunity to play and interact with the cat (e.g., stroking). In this application, the Aamu cat first wanders around, and by swiping the screen, the user can trigger the cat to perform different tricks, such as jumping. Figure 2 presents screenshots of the Arilyn application with Aamu. A detailed description of the application and the integrated AR solution are available at https://www.arla.fi/aamukissa/.
4.2. Case Study 2: Virtual Campus Tour
The second case study was designed and developed by two students at the Haaga-Helia University of Applied Sciences. The aim of this MAR application is to provide a virtual tour of the Pasila campus to those students who received admission to the Business Information Technology (BITE) degree program for the fall 2018 semester. The concept design and implementation processes are out of the scope of this paper.
Figure 3 presents the screenshots of the Virtual Campus Tour application. In this MAR application, an avatar of the BITE study advisor, Riitta, orients new students to the Pasila campus, including information on facilities, lecture rooms, the library and key people. The application helps new students to navigate in the first, third and sixth floors where the classrooms, library and teachers’ and advisors’ offices are located. The avatar helps the student select the correct floor based on the place where they want to visit. For the implementation, we used Unity 3D, 3D videos and Vuforia AR. The Android-based application scans a marker, thus causing the avatar to appear. The student starts to move with the smartphone to get to the door of the elevator where more virtual content is shown. The avatar appears in the front of the elevator next to the buttons, and at the same time, a virtual button panel is displayed. The student is then advised to pick the floor where they want to visit.
Enlarge this image.Figure 3. Screenshots of the Virtual Campus Tour Mobile AR (MAR) application that introduces the Pasila campus to new students.
5. Results
In the following, we present the results of two case study evaluations based on the methods described in Section 3. Then, we collectively summarize the emotions that were identified from the case studies.
5.1. Case Study 1: Aamu Cat
5.1.1. Questionnaire Results
The demographics section of the questionnaire indicated that 15 out of 22 participants had previous experience with MAR, whereas seven participants were inexperienced with MAR, thus suggesting that MAR as a technology has reached a relatively high degree of awareness among young adults.
Figure 4 shows what types of emotions the participants felt towards the Aamu Cat MAR application. The results indicate that positive emotions (relaxing, inspiring, encouraging, exciting) were experienced by most of the participants. Among all participants, a majority (59%) agreed that the MAR application made them relaxed, whereas 14% did not find the MAR application relaxing. Similarly, a majority experienced also other positive emotions, as 73%, 87% and 73% of the participants agreed that the MAR application was inspiring, encouraging and exciting, respectively, with very few negative responses.
Enlarge this image.Figure 4. The participants’ feedback on emotions after the Case Study 1 experiment.
Compared to positive emotions, negative emotions gained much less agreements. Among all participants, 87%, 86% and 77% disagreed that the MAR application was unpleasant, depressing and boring, respectively. Only 9–14% of the participants felt neutral about the negative emotion statements. Although low in proportions, it is noteworthy that 5% and 9% of the participants felt that the MAR experiment was unpleasant and boring, respectively.
5.1.2. Interview Results
In the following, we present the results related to the four interview questions that were asked from the selected participants (9).
Question #1: What was your first impression when you saw the cat?
The participants considered their MAR experiment with the milk package and the Aamu Cat to be interesting and expected to see similar applications of MAR more in public places, for example:
“I really like its simple interactions. Cannot wait for it to kick off in everyday public places.”
(Outi, 18, Female)
Some participants identified the MAR application to be intuitive, thus giving more control to the user on how to utilize the interactions in the application, for example:
“The interaction was engaging and with fun, I almost knew how I should play with the cat.”
(Outi, 18, Female)
Several participants considered that the application engaged them emotionally, which, as per our definition, is the essence of positive user experiences:
“It was emotionally interesting and delightful, and having a real connection with the object, through getting it working took a while.”
(Timo, 27, Male)
“Getting the cat out of the box for the first time was cool […] and its fun for a short time.”
(Anastasia, 28, Female)
“The cat jumping out of the box was surprising.”
(Joan, 21, Female)
The participants’ emotional engagement in the application was also visible in the following terms that frequently appeared in the participants’ comments as their first impressions: unexpected, surprised, liked it, fun, friendly and interactive. In addition to emotional engagement, some participants considered that they became involved with AR and the objects within:
“AR is more interactive, and I feel involved.”
(Mia, 26, Female)
“AR is about feeling the objects.”
(Timo, 27, Male)
Question #2: Did you find it difficult to test the application and bring the cat to the screen?
All students managed to perform the experiment successfully despite no detailed instructions given by researchers. Therefore, all students answered “no” to this question. The main concerns that emerged during the experiment were getting the application to work and that interacting with the cat was time consuming.
Question #3: Would you like to have this type of application in your daily life?
Some participants considered that MAR is suited well to advertisements, buying and selling. The applicability of MAR as a learning technology, however, received more skepticism, for example:
“I’ve tried AR in a language learning application, but I dropped the language course [where the AR learning application was used], so I cannot say whether it is helpful. I prefer the normal two-dimensional (2D) view when it comes to language learning. MAR drains battery way faster.”
(Janne, 24, Male)
The following comment from a participant shows that the perceived benefit for the invested time may depend on the user’s age:
“Maybe kids want to play with cats for minutes if not hours, I do not like to spend time with a virtual pet in my phone.”
(Alexey, 21, Male)
Question #4: How would you change this application?
The changes that almost all participants recommended were having more functionalities in the application, such as: communication with the cat, performing more tricks with the cat and selecting different avatars (e.g., a dog). Furthermore, some participants suggested that the cat’s avatar should be more versatile and realistic and that users should have a choice of personalizing the avatar, as indicated in these comments:
“Choose the figure.”
(Sara, 21, Female)
“Different animal, more colorful surroundings.”
(Mia, 26, Female)
“More entertainment actions.”
(Outi, 18, Female)
“I guess some alert that it would popup is good to have.”
(Timo, 27, Male)
“Feel like this is a beta version, I want to have a more realistic figure. Seems that the cat moves very slow.”
(Ivan, 21, Male)
Furthermore, we discovered that the cat avatar promoted positive UX and may even compensate some factors that contributed to negative UX:
“My first AR experience wasn’t surprising because the app lagged a bit but the cat on class today was really nice.”
(Sara, 21, Female)
5.2. Case Study 2: Virtual Campus Tour
In the usability evaluation, we measured the efficiency of the Virtual Campus Tour application usage, i.e., how well and fast the participants find the locations defined in the tasks with the Virtual Campus Tour application. The loading time of the application was around 0.5–3 s from launching until the participant could start interacting with it. It took on average around 10 min (minimum: 5 min, maximum: 15 min) for the participants to complete the predefined tasks. The participants were generally satisfied with the application’s user interface and the way of interacting with the application.
When prompted in the interviews what the participants would like to change in the application, all of them said that the application should have more instructions of where they should go, the names and locations of places/areas and special places, such as laboratory and computer rooms. Moreover, some participants suggested that the application should also have a map or a list of places in the beginning, so users would know what they can learn from the navigation and where they are in relation to the whole school map. Finally, the navigation could be more playful and interesting by applying a game concept, such as missions to find different areas in the school, and including more interactable objects, as well as background music, to make the tour more relaxing and exciting.
The participants’ main objections concerned the application’s graphics and insufficient control over the application’s performance. For example, the augmented elevator panel did not work properly, as the participants needed to touch a specific area on the floor number to activate it. Although non-active keys were dimmed, some participants still attempted to click them. Moreover, it was not possible to cancel a selected key.
In the UX part, we assessed how the participants felt during and after the experiment. In general, the participants found the application concept to be appealing, especially due to the simple and straightforward interaction and intuitive and obvious movement. The emotions experienced by the participants are listed in Section 5.3.
5.3. Positive Emotions Experienced in the Case Studies
Table 3 summarizes the positive emotions that the participants experienced in our case studies. These emotions were extracted from the data collected by observations, questionnaire (Case Study 1), interviews and the think-aloud method (Case Study 2). It is noteworthy that the diversity of emotions that MAR applications can evoke in users is far beyond those that emerged in this study.
[ Table omitted. See PDF. ]
6. Challenges, Opportunities and Best Practices for MAR UX
The following sections present the challenges, the opportunities and the best practices for MAR UX that emerged from our collective analysis and interpretation of literature and case study findings, as well as from our experiences in developing the Virtual Campus Tour application. Then, we apply some of the identified best practices to create an updated version of the Virtual Campus Tour.
6.1. Challenges of MAR UX
There are many challenges associated with UX design that are specific to MAR applications. Table 4 summarizes some of the challenges that we learned in this study.
[ Table omitted. See PDF. ]
6.2. Opportunities of MAR UX
Despite many challenges, there are significant opportunities that MAR applications bring to UX design. Some of these opportunities, as identified through the case studies, are listed in Table 5.
[ Table omitted. See PDF. ]
6.3. Best Practices for MAR UX Design
As MAR applications seek to create a bridge between the real world and a virtual world, UX designers need to pay attention to the special requirements that arise from the interplay between the two worlds. To alleviate this challenge, we have devised a set of best UX design practices for MAR applications based on the analysis of the case studies. These practices are listed in Table 6, together with descriptions and examples. By following these practices, UX designers can increase the likelihood of users becoming emotionally attached to the target MAR application. It is noteworthy that this list is intended to complement, not replace, existing mobile UX design guidelines.
[ Table omitted. See PDF. ]
6.4. Applying Best Practices to Virtual Campus Tour
Figure 5 presents an updated version of the Case Study 2 application (Virtual Campus Tour) based on some of the best practices presented above. First, we designed a new avatar for Riitta, whom the freshmen meet when they arrive at the university for the first time. We designed the avatar to be as realistic as possible, thus providing a true reality-virtuality object correspondence and object detail to new students. It is not merely a design transformation from a 2D to 3D object; the MAR object actually represents a real-world entity, in this case a person. Hence, we used an existing and expected metaphor for MAR content representation.
Furthermore, to make MAR objects more realistic, all of them were resized to match their real sizes. To further improve spatial correspondence, we ensured that the locations of MAR objects were aligned with their real-world counterparts, where applicable. This is unlike in the original Virtual Campus Tour, where the virtual panel size and location were far from those of the actual panel.
Furthermore, to engage users emotionally with the MAR application through natural interaction, we enabled voice commands. The student can start to communicate with the avatar by saying “Hello” as soon as the avatar appears. The avatar responds, and then the student may ask, for example: “Who are you?” The avatar first introduces herself, and then, she explains about the different floors, classrooms and offices. Finally, after the tour ends, the avatar wishes the student all the best. A future study is needed to measure whether the added voice commands and other changes to the avatar make it more emotion-evoking and natural.
7. Discussion
With the advancement of smart device technologies and the popularity of smartphones, users’ expectations have exceeded mere usability of mobile applications. In contemporary application design and development, designers must not only expect that the application is easy and efficient to use while satisfying the user’s essential needs, but that it also makes the user emotionally engaged. Emotional engagement is associated with the experience that the user constructs while interacting with the application. Yet, UX is a complex concept, as it deals with many different factors. The focus of this paper was on the emotional aspects of UX in the context of MAR applications. Furthermore, MAR UX design, development and evaluation are somehow different than those of traditional mobile applications. From the MAR concept design perspective, UCD frameworks, such as mLUX (m-Learning User eXperience) [30], provide relevant concept development methods that ensure appropriate usability together with UX factors, including, but not limited to delightfulness, reliability and adjustability.
From the development perspective, various integrated development environments, such as Unity 3D and Visual Studio, and AR toolkits, such as Vuforia AR, provide great assistance to developers to come up with MAR applications. In fact, there even exist AR toolkits (e.g., ARIS, Augment, Aurasma, Blibbar and Layar) that enable the creation of full MAR experiences without programming skills [3]. Consequently, in the wake of the success of Pokémon Go, the number of published MAR applications is soaring, but research is needed to determine to what extent they engage users emotionally.
Unlike the MAR concept design and MAR application development, MAR UX design and assessment are associated with many challenges (Table 4) that require further research and tool development. This has also been identified previously by other researchers who have investigated users’ overall impressions of MAR applications. For example, some Pokémon Go players believe that the AR feature of the game drains the smartphone, which may lead them to disable the AR feature [18,19]. Furthermore, MAR applications often require bodily involvement beyond mere touching. For example, users may need to walk to the correct location to launch a location-aware MAR experience, orient the device appropriately in relation to the real world to see the target MAR scene, use hand gestures to control MAR objects in the scene and possibly use voice commands or movement gestures to further interact with the application. If done poorly, this rich interaction can make the application unfriendly, difficult to use and therefore emotionally disengaging.
The interaction with MAR applications in public spaces requires special design considerations. It is not enough to make the application easy to use, which is a de facto design goal for all mobile applications, including MAR applications. Designers of mobile applications should acknowledge that, for safety reasons, the application should not disrupt users’ awareness of their surroundings. This is even more important in the case of MAR applications that often require deeper interaction than ordinary mobile applications. Moreover, Carmigniani et al. [44] believe that the main problem regarding social acceptance comes from the level of disruption caused by mobile devices (in other words: mobile applications) in public places and during conversations.
The acceptance of MAR applications from the UX perspective requires that the application follows the user’s mental models. For example, in Case Study 1, an interviewed participant mentioned that she prefers dogs over cats. Consequently, she did not find the cat as appealing as some other participants did. This indicates that designers must put attention on the correspondence of selected metaphors and mental models of the target users. The best way to achieve this is to utilize a UCD method that involves end-users as co-designers, thus revealing their mental models in the design process.
The acceptance of MAR also depends on whether users are sufficiently skilled to perform the necessary application setup and configuration tasks, which should be minimal. For example, the participants of Case Study 1 had to install the Arilyn application before interacting with the cat. Moreover, the results demonstrated that the participants, who were university students, were ready to use MAR applications if and only if they find an added-value in having the application installed in their smartphones. This conclusion is only valid in the context where the experiment was conducted; in other contexts, and with other users, the results could be different.
There are several limitations in this study that are noteworthy. Firstly, the sample sizes in case studies was relatively small; thus, we cannot draw statistically-significant conclusions. Statistical analysis of a large quantity of interview data could, for example, be used in a future study to identify emotions that are particularly prominent among MAR application users. Secondly, because the evaluation experiments were conducted in a single session, long-term effects of UX design remain to be investigated. For example, it would be important to investigate how long a MAR application succeeds in supporting emotional engagement of its users, as it directly relates to sustainable use of the application. Thirdly, the interviews were fully structured, thus leaving little room for drilling deeper into specific topics of interest. Considering these shortcomings and the results of this study, we aim to develop a richer MAR application and investigate its UX with a larger sample size and versatile data collection instruments over a long period of time in the future.
8. Conclusions
The aims of this study were to find out how users perceive MAR applications emotionally, to identify challenges and opportunities associated with MAR applications and to recommend best practices from the perspective of MAR UX design. These aims were reached through evaluations of two case studies: a commercial advertisement application featuring an animated AR cat and our Virtual Campus Tour application designed to orient new students at a university campus. The analysis of the evaluation results, together with the conducted literature review and the experiences acquired during the development of the Virtual Campus Tour application, were the basis for identifying several challenges, opportunities and best practices for MAR UX. The identified best practices were then used to update the Virtual Campus Tour application, thus increasing its potential to engage users emotionally in the future. Whether this potential is realized or not remains to be evaluated in a future study.
The evaluations of the two case studies demonstrated that the participants value MAR applications based on their functionalities, usefulness and ease of use. A major reason as to why designers should utilize AR in mobile applications is to provide positive experiences to users by engaging them emotionally with the application. Indeed, as we have shown in this study, emotional engagement is closely associated with the term user experience. We further demonstrated that having AR objects embedded in a mobile application requires proper design considerations, such as identifying users’ needs, providing natural interaction and engaging users with the application through a realistic avatar.
The results of this study may help UX designers come up with more user-friendly and robust MAR applications. Yet, there remain many research gaps to be investigated, such as developing an acceptance model for MAR UX, modeling MAR content interaction, personalizing MAR experiences and sustainable use through appropriation of MAR experience lengths. We also recommend further studies on MAR applications in various contexts to draw universal MAR UX guidelines based on the best practices presented here.
Author Contributions
A.D. designed and performed the experiments. A.D. and T.H.L. analyzed the data. A.D. and T.H.L. wrote the paper.
Acknowledgments
Special thanks go to the students at the User Experience Design and User Experience Project courses in spring 2018 at the Haaga-Helia University of Applied Sciences.
Conflicts of Interest
The authors declare no conflict of interest.
1. Scholz, J.; Smith, A.N. Augmented reality: Designing immersive experiences that maximize consumer engagement. Bus. Horiz. 2016, 59, 149–161.
2. Kipper, G.; Rampolla, J. Augmented Reality: An Emerging Technologies to Guide AR; Elsevier: Waltham, MA, USA, 2013.
3. Laine, T.H. Educational Mobile Augmented Reality Games: A Systematic Literature Review and Two Case Studies. Computers 2018, 7, 28.
4. Bacca, J.; Baldiris, S.; Fabregat, R.; Graf, S.; Kinshuk, D. Augmented Reality Trends in Education: A Systematic Review of Research and Applications. Educ. Technol. Soc. 2014, 17, 133–149.
5. Lee, B.K. Augmented Reality in Education and Training. TechTrends 2012, 56, 13–21.
6. Radu, I. Augmented reality in education: A meta-review and cross-media analysis. Pers. Ubiquitous Comput. 2014, 18, 1533–1543.
7. Wu, H.; Lee, S.W.; Chang, H.; Liang, J. Current status, opportunities and challenges of augmented reality in education. Comput. Educ. 2013, 62, 41–49.
8. Dunleavy, M.; Dede, C. Augmented Reality Teaching and Learning. In Handbook of Research on Educational Communications and Technology; Spector, J., Merrill, M., Elen, J., Bishop, M., Eds.; Springer: New York, NY, USA, 2014; pp. 735–745.
9. Cheng, K.-H.; Tsai, C.-C. Affordances of Augmented Reality in Science Learning: Suggestions for Future Research. J. Sci. Educ. Technol. 2013, 22, 449–462.
10. Nielsen, B.L.; Brandt, H.; Swensen, H. Augmented Reality in science education—Affordances for student learning. Nord. Stud. Sci. Educ. 2016, 12, 157–174.
11. Fotaris, P.; Pellas, N.; Kazanidis, I.; Smith, P. A systematic review of Augmented Reality game-based applications in primary education. In Proceedings of the 11th European Conference on Game Based Learning, Graz, Austria, 5–6 October 2017; pp. 181–191.
12. Ahonen, T. Tomi Ahonen Calls Out: Augmented Reality is the 8th Mass Medium. VR World. 2012. Available online: http://vrworld.com/2012/04/11/tomi-ahonen-calls-out-augmented-reality-is-the-8th-mass-medium/ (accessed on 20 May 2018).
13. Billinghurst, M.; Clark, A.; Lee, G. A Survey of Augmented Reality. Pres. Teleoper. Virt. Environ. 2015, 8, 73–272.
14. Mekni, M.; Lemieux, A. Augmented Reality: Applications, Challenges and Future Trends. In Proceedings of the 13th Applied Computer and Applied Computational Science, Kuala Lumpur, Malaysia, 23–25 April 2014; pp. 205–214.
15. Althoff, T.; White, R.W.; Horvitz, E. Influence of Pokémon Go on Physical Activity: Study and Implications. J. Med. Internet Res. 2016, 18, e315.
16. Tateno, M.; Skokauskas, N.; Kato, T.A.; Teo, A.R.; Guerrero, A.P.S. New game software (Pokémon Go) may help youth with severe social withdrawal, hikikomori. Psychiatry Res. 2016, 246, 848–849.
17. Dirin, A. From Usability to User Experience in Mobile Learning Application; Aalto University: Helsinki, Finland, 2016; p. 316.
18. Paavilainen, J.; Korhonen, H.; Alha, K.; Stenros, J.; Koskinen, E.; Mayra, F. The Pokémon GO Experience. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems—CHI ’17, Denver, CO, USA, 6–11 May 2017; pp. 2493–2498.
19. Rasche, P.; Schlomann, A.; Mertens, A. Who Is Still Playing Pokémon Go? A Web-Based Survey. JMIR Ser. Games 2017, 5, e7.
20. Ritsos, P.; Ritsos, D.; Gougoulis, A. Standards for Augmented Reality: A User Experience perspective. Stand. Meet. 2011, 17, 1–9.
21. Olsson, T.; Lagerstam, E.; Kärkkäinen, T.; Väänänen-Vainio-Mattila, K. Expected user experience of mobile augmented reality services: A user study in the context of shopping centres. Pers. Ubiquitous Comput. 2013, 17, 287–304.
22. Roto, V.; Law, E.; Vermeeren, A.; Hoonhout, J. User Experience White Paper—Bringing Clarity to the Concept of User Experience. Available online: http://www.allaboutux.org/files/UX-WhitePaper.pdf (accessed on 20 May 2018).
23. Tokkonen, H.; Saariluoma, P. How User Experience is Understood? In Proceedings of the Science and Information Conference (SAI), London, UK, 7–9 October 2013; pp. 791–795.
24. Carlos, J.; Nicolás, O.; Aurisicchio, M. A Scenario of User Experience. Design 2011, 1, 1–12.
25. Hassenzahl, M. User experience (UX): Towards an experiential perspective on product quality. In Proceedings of the 20th International Conference Association Francophone d’Interaction Homme-Machine—IHM ’08, Metz, France, 2–5 September 2008; pp. 11–15.
26. Hassenzahl, M.; Tractinsky, N. User experience—A research agenda. Behav. Inf. Technol. 2006, 25, 91–97.
27. Yong, L.T. User experience evaluation methods for mobile devices. In Proceedings of the Third International Conference on Innovative Computing Technology, London, UK, 29–31 August 2013; pp. 281–286.
28. Väätäjä, H.; Roto, V. Mobile questionnaires for user experience evaluation. In Proceedings of the 28th of the international conference extended abstracts on Human factors in computing systems—CHI EA ’10, Atlanta, GA, USA, 10–15 April 2010; p. 3361.
29. Pelet, J.-É.; Taieb, B. Enhancing the Mobile User Experience Through Colored Contrasts. In Encyclopedia of Information Science and Technology, 4th ed.; 2018; pp. 6070–6082.
30. Dirin, A.; Nieminen, M. mLUX: Usability and User Experience Development Framework for M-Learning. Int. J. Interact. Mob. Technol. 2015, 9, 37–51.
31. Irshad, S.; Rohaya Bt Awang Rambli, D. User experience of mobile augmented reality: A review of studies. In Proceedings of the 2014 3rd International Conference on User Science and Engineering (i-USEr), Shah Alam, Malaysia, 2–5 September 2014; pp. 125–130.
32. Arth, C.; Grasset, R.; Gruber, L.; Langlotz, T.; Mulloni, A.; Wagner, D. The History of Mobile Augmented Reality. Comput. Graph. Vis. 2015, 2, 1–40.
33. Prochazka, D.; Stencl, M.; Popelka, O.; Stastny, J. Mobile augmented reality applications. In Proceedings of the Mendel 2011 17th International Conference on Soft Computing, Brno, Czechoslovakia, 15–17 June 2011; pp. 469–476.
34. Rashid, R.A.; Mohamed, H.; Hussin, A.R.C. Mobile Augmented Reality Tourism Application Framework. In Recent Trends in Information and Communication Technology: Proceedings of the 2nd International Conference of Reliable Information and Communication Technology (IRICT 2017); Springer: Berlin, Germany, 2018; pp. 108–115.
35. Olsson, T.; Väänänen-Vainio-Mattila, K. Expected User Experience with Mobile Augmented Reality Services. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, Stockholm, Sweden, 30 August–2 September 2011; pp. 1–4.
36. Jung, T.; Chung, N.; Leue, M.C. The determinants of recommendations to use augmented reality technologies: The case of a Korean theme park. Tour. Manag. 2015, 49, 75–86.
37. Kim, M.J. A framework for context immersion in mobile augmented reality. Autom. Constr. 2013, 33, 79–85.
38. Irshad, S.; Awang Rambli, D.R. Multi-layered Mobile Augmented Reality Framework for Positive User Experience. In Proceedings of the The 2th International Conference in HCI and UX on Indonesia 2016—CHIuXiD ’16, Jakarta, Indonesia, 13–15 April 2016; pp. 21–26.
39. Agarwal, A.; Sharma, N.K.; Gupta, P.; Saxena, P.; Kumar Pal, R.; Mehrotra, S.; Nair, P.; Wadhwa, M. Mobile Application Development with Augmented Reality. Int. J. Comput. Sci. 2014, 2, 20–25.
40. Nielsen, J. How many test users in a usability study. 2012. Available online: https://www.nngroup.com/articles/how-many-test-users/ (accessed on 20 May 2018).
41. Falk, E.B.; O’Donnell, M.B.; Tompson, S.; Gonzalez, R.; Dal Cin, S.; Strecher, V.; Cummings, K.M.; An, L. Functional brain imaging predicts public health campaign success. Soc. Cogn. Affect. Neurosci. 2016, 11, 204–214.
42. Saldana, J. An Introduction to Codes and Coding. In The Coding Manual for Qualitative Researchers; SAGE Publications: London, UK, 2015; p. 368.
43. Davison, G.C.; Vogel, R.S.; Coffman, S.G. Think-aloud approaches to cognitive assessment and the articulated thoughts in simulated situations paradigm. J. Consul. Clin. Psychol. 1997, 65, 950–958.
44. Carmigniani, J.; Furht, B.; Anisetti, M.; Ceravolo, P.; Damiani, E.; Ivkovic, M. Augmented reality technologies, systems and applications. Multimed. Tools Appl. 2011, 51, 341–377.
1Business Information Technology, Haaga-Helia University of Applied Sciences, 00520 Helsinki, Finland
2Department of Computer Science, Electrical and Space Engineering, Luleå University of Technology, 93187 Skellefteå, Sweden
*Author to whom correspondence should be addressed.
© 2018. This work is licensed under http://creativecommons.org/licenses/by/4.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.