Evaluating the Benefits of Collaborative VR Review for Maintenance Documentation and Risk Assessment

Abstract

Featured Application

Based on data from a globally operating industrial company, this study demonstrated the benefits of VR to maintenance documentation review and risk assessment processes.

Abstract

Technical documentation creation is a collaborative process involving several departments in R&D. Even though virtual reality (VR) has been demonstrated to facilitate industrial collaboration and advance the product development lifecycle in earlier studies, it has not been utilized for technical documentation review and risk assessment processes in industrial companies. This article presents a case study where the benefits of VR to maintenance documentation reviews and risk assessments were studied. The virtual reality environment was tested by nine domain experts from an industrial company in a user study that replicated their actual real-life industrial collaboration tasks. Both qualitative and quantitative data were collected during the study. Our findings show that collaborative VR has the potential to enhance the documentation review and risk assessment processes. Overall, the concept of using virtual reality for documentation review and risk assessment processes was rated positively by participants, and even though further development is needed for the review tools, VR was viewed as a concept that facilitates collaboration, enhances the current review practices, and increases spatial understanding. The benefits of VR are evident, especially for geographically scattered teams that rarely meet face-to-face or do not have access to the actual physical equipment. In cases where traditional means of communication are not enough, process improvements are needed for documentation review and risk assessment processes, and our proposed solution is VR.

1. Introduction

For many industrial companies, the maintenance business is growing in importance, and more focus is paid to providing support and technical instructions to the maintenance technicians on the field. Industrial maintenance tasks are often complicated, and technicians need instructions to perform the tasks in a safe and efficient manner.

KONE Corporation is a global leader in the elevator and escalator industry [1]. KONE operates in more than 60 countries with approximately 30,000 field employees. KONE publishes hundreds of new or revised maintenance instructions each year to support its service business. As the safety and accuracy of technical instructions are essential to the company, there is no room for ambiguity or misunderstandings in the instructions delivered to the field. To achieve this, KONE has been developing both the practices and processes for maintenance instruction creation and the digital channels for technical information delivery to field employees.

Technical documentation creation is a collaborative process involving several departments, including technical documentation, subject matter, and risk assessment experts. While the technical documentation experts prepare the drafts, the subject matter and risk assessment experts validate the content in reviews. Both the preparation of the draft and its review are equally important in the development of a technical instruction.

The review of maintenance instructions calls for the collaboration of technical documentation, maintenance method development and risk assessment. Preferably, this review is performed so that all the parties are in the same location and have access to both the equipment and instructions. However, due to the lack of access to the physical equipment and difficulties in remote communication between globally distributed teams and departments, these reviews are in many cases carried out by technical documentation experts sending out PDF files or links to review portals and subject matter experts commenting on them remotely. This trend was further accelerated by the COVID-19 pandemic and related restrictions when people from the same location were also forced to work remotely from home. Therefore, instead of an interactive collaborative process, the review becomes a process where the parties work in isolation. Because the work is carried out individually, the reviewers might send conflicting review comments, or some questions might be left unanswered altogether.

In most of the existing technical communication literature and guidelines used in companies, documentation reviews are discussed at a fairly abstract level and the focus is on different types of checklists, tips, and best practices. For example, Hackos and Jayaprakash discuss the importance of reviews, both technical reviews and peer reviews, their effect on the quality of documentation, and the parties that should be involved in the review process [2,3]. However, the technical communication literature does not go into detail on how reviews should be physically arranged, especially in globally operating companies. Similarly, risk assessment is generally guided by regulations that do not discuss the actual best practices of risk assessments or the physical setup.

Some research has been conducted for evaluating the use of novel technologies, such as virtual reality (VR) for the technical documentation creation process [4], but their use has not been implemented in practice in industrial companies. Furthermore, even though the use of VR has proven efficient for engineering design review [5], VR has not, until now, been studied or utilized for technical documentation reviews and risk assessments, processes that have much in common with engineering design reviews. As VR has been proven to be an effective tool to aid collaboration and cooperation [6], it would also be a good fit for technical documentation reviews and risk assessments as both processes are inherently very collaborative.

The work described in this paper contributes to the research in the fields of virtual reality applications in industrial systems and technical communication. The study investigates the potential of VR as a collaborative review and risk assessment platform and specifically addresses the following research question:

RQ: What are the benefits of collaborative virtual reality to maintenance documentation reviews and risk assessments in industrial companies?

To address the research question, we conducted user tests with domain experts that are representative of the intended users of the VR system. A total of nine users from KONE Corporation tested the VR environment and gave their feedback on the usefulness and benefits of the system for technical documentation reviews and risk assessments. The tasks performed during the user testing were designed to replicate an actual review and risk assessment of a maintenance method and instructions related to a product from KONE. Even though this study is focused on a single industrial company’s documentation and risk assessment processes, the documentation review and risk assessment are universal collaborative processes that are very similar in other industrial companies. Therefore, this case study is representative of the generic maintenance documentation review and risk assessment processes used in many industrial companies.

2. Background

In this section, we firstly introduce VR as a technology to facilitate collaboration and demonstrate its application cases and resulting advantages in the industrial context. Then, we provide a more detailed background for areas that are relevant to this case study, including industrial maintenance, the maintenance documentation process, and risk assessment. In industrial maintenance, it is essential that maintenance tasks are performed in an optimal and safe way, and maintenance methods are, therefore, carefully designed, authored, tested, documented, and risk assessed. Even though these processes in industrial companies’ R&D are inherently collaborative in nature, collaborative VR has not, until now, been utilized to enhance these processes in industrial companies.

2.1. Collaborative VR

The application of VR to industrial needs has been investigated for several decades, showing the potential to aid, enhance and transform many of industrial tasks and processes [7,8]. With a given flexibility to simulate dangerous contexts and enable natural interactions with virtual objects in immersive virtual environments [7,9,10,11], VR has been successfully applied in industry to facilitate training [12,13], AR-prototyping [14,15] as well as different phases of product development cycle [7,16,17,18,19].

“Distributed virtual environments” [20], also widely referred as collaborative VR (CVR), have become especially in demand during the COVID-19 pandemic, when people all over the globe were forced to work remotely. The major advantage of collaborative VR is the possibility to blur geographical barriers and immerse people from diverse locations into shared working spaces [21], addressing the needs of multidisciplinary global collaboration. Due to the increased feel of presence and immersion together with the ability to communicate verbally and non-verbally, collaboration in VR is understood as more efficient and flexible than the collaboration via traditional conferencing tools. Evidence has shown that VR is capable of positively affecting the elements of remote communication, such as the clarity and richness of communicated information, and enhance the quality of discussion and knowledge transfer due to shared context and awareness of others [22,23,24].

A case study by Berg et al. [16], for instance, demonstrated the application of VR to support early design decision making, which resulted in escalation of identified design issues and provided solutions, in addition to increased sense of team engagement and participation in the collaboration process. Furthermore, a recent study demonstrated that real-time collaboration over multi-user VR leads to increased performance in comparison to the traditional approach [25]. The studies by Wolfartsberger et al. [17,26], which explored the use of VR to aid the collaborative design review process, concluded that VR technology is a “useful addition, and not a replacement”, which potentially accelerates the process and ensures inclusion of all professional groups. Other studies reported that collaboration in VR may strengthen lean and agile practices [27,28,29,30], optimizing value creation and team performance, while reducing resource waste and time span. Furthermore, collaboration in VR supports the innovation mindset of employees and overall sustainability [31], whereas VR itself is recognized as motivating and engaging technology by industrial employees [12,14].

2.2. Industrial Maintenance and Maintenance Documentation

Industrial maintenance aims at keeping machinery running and in good condition. Companies have different maintenance strategies; in the era of data analytics, the trend is towards preventive and condition-based maintenance. Complete optimization of material and workforce costs both per visit and over the equipment lifecycle, increasing equipment uptime, and avoiding risk of breakdown have transformed the nature of maintenance visits. Where earlier it was typical to have predetermined maintenance visits with predefined task lists, modern service companies use real-time sensor data to monitor the condition of the machinery and artificial intelligence to define the optimal time for each maintenance task to be performed. This means that the content of each maintenance visit is different, and the composition of tasks varies across the visits. Thus, maintenance technicians need instructions on what to do as they cannot rely on their experience or tacit knowledge as before. Because of the quest for financial optimization, it is equally important to instruct what not to do on the visit.

Regardless of the maintenance strategy and how the composition of tasks for each maintenance visit is determined, it is important that maintenance tasks are performed in an optimal and safe way. Therefore, maintenance methods are carefully designed, authored, tested, documented, and risk assessed.

Technical communication is a field that conveys technical or specialized information, uses technology to communicate, or provides instructions on how to do something [32]. The maintenance documentation process is a subcategory of the more generic technical documentation process [33,34], and the outcome of the maintenance documentation process is a set of maintenance instructions that help the end users, maintenance technicians, complete their tasks in an efficient and safe manner. The content creation process is inherently collaborative, where people from different departments are working together to achieve a common goal. The process starts from maintenance method and outline creation by maintenance method developers. The outline is then passed on to the technical documentation experts, who start working on a draft. The draft instructions are developed iteratively with the maintenance method developers, by reviewing and revising. When both parties are satisfied with the draft, the maintenance method and the safety of the instructions are evaluated with the help of risk assessment experts. If the risk assessment finds deficiencies in the instruction or the method behind the instruction, they are revised and reviewed again until the requirements for safety are satisfied. Finally, the instructions are officially checked and approved by the organization, and then published into relevant delivery channels. See Figure 1 for an overview of the maintenance documentation process.

From the technical point of view, the process of documentation review has remained the same for the past years. Even though the use of VR for the technical documentation process has been studied [4], the use of novel technologies, such as VR, has not been implemented for the review and risk assessment of technical instructions in industrial companies. Typically, technical instruction reviews and risk assessments are carried out by sending out links to PDF files or online review portals. If all participants are located on the same site, a face-to-face meeting can be arranged, but global teams rarely have the option of doing this. In practice, meetings are held in conferencing tools, such as Microsoft Teams, or, more often, reviewers comment on the PDF file and send it back to documentation experts via email or file sharing systems. In many cases, the teams have to work without any access to the actual product. As the development cycle in industrial companies is short, the technical instructions have to be completed in an increasingly short time frame, often before any actual prototypes exist [4]. Furthermore, even if a prototype exists, it is usually located on one site only and not accessible to everybody, especially in the case of globally scattered teams. In a conference call, even when a 3D model is shown via screen share, spatial understanding is missing and explaining product proportions and features is difficult, if not impossible. To combat these problems, process improvements are needed for documentation review and risk assessment, and our proposed solution to these challenges is VR.

2.3. Risk Assessment and Codes and Standards

The lifetime use of machinery, including the phases of transport, assembly, operation, adjustment, maintenance, dismantling, disabling and scrapping, must be safe. This is a legal requirement in, for example, European regulations [35,36].

While the regulations and, for example, the European harmonized standard EN 13015 [37] for maintenance instructions of lifts and escalators do not give explicit requirements when and how the safety risk assessment should be carried out, there are certain standards to follow. ISO 12100 [38] gives generic guidance on how a risk assessment should be carried out. For lifts under lifts directive and for escalators under machinery directive, there is an international standard ISO 14798 [39] that provides a well-defined risk assessment methodology to follow.

When creating solutions or services, risk assessments can be performed in VR in different phases of the solution creation process, from assessing the initial concept to assessing final designs with prototype and piloting feedback. There are studies on risk assessing designs from user safety point of view [40,41], some concentrating on ergonomics [42].

The use of VR has also been studied for training [43]. The psychological risk-taking decision process is discussed in de-Juan-Ripoll et al.’s work [44], recommending that VR enhanced with physiological measurements is further studied for assessing attitudes to risk, risk perception, and conditioning factors. Using VR for delivering safety training has been studied in Leder et al.’s work [45], considering the impacts of VR on risk perception, learning, and decision making.

To build on these, technical documentation needs to provide accurate information for a solution or service, including important safety related information. Thus, technical documentation itself can be considered a subject for risk assessment. When risk assessing technical instructions, risk assessment experts check the tasks described in the instructions for any unsafe methods. They sometimes also request that warnings are added to the instructions to mitigate risks. They might recommend a different way of performing a task or safety measures that need to be carried out before or after the task to prevent injury to people or damage to equipment. After the risk assessment, the instructions are deemed to be safe to be published and used by field employees.

Even though the use of VR has been studied for several areas of risk assessment, until now, there are no studies that evaluate the benefits of VR to risk assess the contents of technical instructions.

3. Materials and Methods

In this section, we detail an exploratory user study, which is the third iteration round of a project that investigates the application of VR for technical documentation creation purposes. The previous iterations were focused on the early phases of documentation creation [46,47,48], while in this work, we demonstrate how VR can be used for collaborative technical documentation reviews and risk assessments. The focus of this study is not on the development of a VR platform but evaluating the usefulness and benefits of VR for the case study.

3.1. Methodological Proposal—Workflow

The aim of our study was to investigate the application of collaborative VR in an area where it has not studied earlier, maintenance documentation review and risk assessment. Both of these processes are very collaborative but in reality, especially in globally operating companies, people involved often do not get to meet face-to-face or have access to the equipment, which complicates the processes.

We propose that collaborative VR is a good fit to tackle these challenges in technical documentation reviews and risk assessments. Instead of meeting in conference calls or working individually by studying 3D models from computer screens and commenting on PDF files, we propose that the experts involved in the process use collaborative VR to meet each other, study the virtual equipment, demonstrate the maintenance method, review the related maintenance instructions, and assess if there are any risks involved in the tasks described in the instructions.

3.2. Implementation: VR Platform for Industrial Collaboration

The VR platform used in the user study, COVE-VR, was designed based on the input of subject matter experts and evaluated in collaboration between industrial and academic researchers [46] in several iterations and scenarios [46,47,48]. The following two virtual environments (Lab and Showroom) were deployed to facilitate a wide scope of industrial tasks: (1) a small-sized Lab replicates the realistic context of an elevator shaft based on a 3D CAD model and (2) a Showroom is a larger space to facilitate collaboration of multiple multidisciplinary teams and in-depth investigation of 3D CAD models that would not be possible to perform in a smaller space. To accomplish industrial tasks in VR, the users can utilize virtual tools that are opened from a wrist menu, which is the main menu of the platform. It is visualized as a circle menu around the user’s wrist and opened when the user is hovering their finger over the controller’s touchpad. When a user is opening their wrist menu, the menu is not visible to other users, but any virtual tool opened from the menu is visible to all users working in the VR. Additionally, all the components in the VR environment are visible to all users.

The users are visualized as simplistic avatars with a name label as shown in Figure 2; the voice icon appears over the avatar when the user is talking. Users can view each other as avatars, working in the VR and interacting with the components in VR. The users are able to locomote in the VR by moving in the physical space or by teleporting in the VR environment.

In this article, we provide a description of the virtual tools that are available in the VR environment used in the study and relevant for the “review of technical documentation” scenario. First of all, technical documentation can be opened in VR with the DocPanel tool; it reads XML files and visualizes maintenance methods and technical instructions in the form of text and graphics over a floating window that users can view and control in the VR (see Figure 2). The XML files have been created in the company content management system (CMS), exported from the CMS, and stored in the VR computer. The instructions can then be loaded to the DocPanel from a menu. Once the instruction is loaded to the DocPanel, it shows the task step by step, and the user may jump between pages, move the panel freely in the virtual environment, and place it in a comfortable spot for the users immersed in the VR. The concept of the DocPanel as a floating window was adopted from a preceding study [14], where it was used to visualize maintenance instructions for in-field AR guidance and to test it in VR. The DocPanel tool is available to all users in the VR. Once a user has opened the instruction in the DocPanel, all the users in VR can view it and control the DocPanel and its functionalities.

In addition, previously tested [46,47,48] TextBox, Camera and Measure tools can be used as supportive tools during the review. With the TextBox tool, a user can input text via speech recognition (English or Finnish) or a virtual keyboard. The camera tool can be used to take pictures and videos in the VR environment, and the measure tool to take measurements of the dimensions and distances of the components in the VR environment. All generated digital content (e.g., text sequence, pictures, and videos) are saved to a storage folder, which can be further accessed via the desktop.

3.3. Case Study

This article describes a case study that was designed to test the usefulness and benefits of virtual reality for the maintenance documentation review and risk assessment process. The goal of the study was to explore if collaborative VR can enhance the company processes and collaboration of globally scattered teams. To address the research question, the VR scenario was tested by domain experts of KONE Corporation, a globally operating leader in the elevator and escalator industry, in a user study that replicates the actual real-life industrial collaboration tasks related to maintenance documentation reviews and risk assessments, both at KONE and other industrial companies.

3.3.1. User Study Procedure and Task Description

The virtual reality environment was tested by participants from three different departments that collaborate in the maintenance documentation process, including documentation, maintenance method development, and risk assessment. In the user test, each of the participants had their own dedicated role related to the actual department they work in at the company. The test participants were requested to work with a maintenance instruction displayed in the DocPanel tool (see Figure 3) and interact with the components in the VR environment. The instruction described a battery replacement task for a component, involving the removal of the battery and its cabling, installation of a new battery and reconnection of its cables and all the safety information related to the tasks. The documentation expert was responsible for leading the review, controlling the DocPanel tool and taking notes with the TextBox tool. Both virtual keyboard and speech-to-text functionalities were available to the users, and they could use both features according to their preferences. The maintenance method expert acted as the subject matter expert, reviewing the technical details of the instructions, and clarifying any open issues with the component or the maintenance method. The maintenance method export demonstrated the maintenance method by, for example, opening the cover of the component and removing the battery. The documentation and maintenance method expert then reviewed the draft instructions and noted down any missing information or need for additional illustrations to be generated. When the parties agreed that an additional illustration should be added, the documentation expert used the camera tool to take a picture to help with the creation of the illustration that would take place after the review. While the other parties reviewed the technical correctness of the instructions, the risk assessment expert reviewed the safety of the working environment and the maintenance method. If the risk assessment expert noticed any deficiencies in the safety information in the reviewed instructions, they commented on it to the other users, and the users then proceeded to discuss what type of warnings, for example, would need to be included in the instructions. The documentation expert then noted down the final decision of what needs to be added with the TextBox tool.

During the test, the participants were encouraged to interact with each other and utilize the tools available in the VR environment (TextBox, Camera and Measure). The participants were also encouraged to comment on the functionalities of the tools and their suitability for the tasks they were responsible for. Thinking aloud and participant observation were used as the methods for collecting the data. The participants were asked to think aloud while performing the tasks, thus enabling the observers to understand what they liked and disliked. Thinking aloud also made it clear to the observers if the participants had trouble using the system or understanding some functionalities. The sessions were video recorded so that observers could go back and check details after the user tests if needed.

3.3.2. User Study Setup

In each session, three user study participants were located in different rooms, each wearing a VR head-mounted display. Two HTC Vive and one HP Reverb VR sets were used. One on-site facilitator was present in each of the three rooms to provide assistance and ensure the safety of the participants. The user study procedure was moderated by one of the on-site facilitators via Microsoft Teams on a laptop, connecting with the participants, asking them to accomplish the tasks, and encouraging their full participation in the tests. Teams established an audio connection between the rooms; the audio from the VR sets was muted so there was no interference with the audios. Teams also streamed the video of the physical space of the main facilitator and the participants from two rooms. The VR view from one user test participant was also streamed to Teams; the VR computer was used for sharing this stream.

The user study was observed by three observers; two were present on site observing the participants, and one was observing the procedure remotely with the Teams video stream. Observers were watching for certain behaviors and taking notes on the things that they observed the participants doing. The user study setup and the view of the Teams streams can be observed in Figure 4.

The user study participants were immersed in VR, and the Teams stream was not visible to them; it was only used for audio and observation purposes. The Teams sessions were recorded for future reference. Figure 5 and Figure 6 show user tests in progress.

The test setup, researchers’ roles as well as the instructions and tasks for the participants were tested in a separate pretest session. Based on the learnings from the pretest, some modifications were made. For example, in the VR environment, the participant names and department were added above each avatar to make it easier to recognize participants in VR. Some details of the tasks for the participants were also modified to bring clarity to the test sessions. After the pretest, no modifications were made between actual user tests; thus, they were all equal.

3.3.3. User Study Participants

Three user test sessions of the collaborative review process were held in the COVE-VR platform. In each of the sessions, we had three participants, i.e., one subject matter expert from each of the departments involved, including maintenance method development, technical documentation, and risk assessment. This makes a total of nine experts (aged from 34 to 64 (M = 49); seven males and two females). All participants had a university degree, six bachelors and three master’s degrees. On average, their experience at their role was 9.5 years, with a minimum at 2 and maximum at 21 years. Four experts had already been included in the process of testing COVE-VR in earlier studies; two of them had participated in all of the iterations and two were partly involved. Our test participants were carefully selected as they have high domain-specific expertise from the three fields; therefore, their opinions carry considerable weight for evaluating the benefits of collaborative VR for documentation review and risk assessment.

3.3.4. Collected Data and Analysis

Both qualitative and quantitative data were collected during the study. The quantitative data were collected via pre and post online surveys, created with the LimeSurvey tool. The validated evaluation method SUXES was used to collect user experience and analyze differences in expectations and actual experiences with the VR environment [49]. As SUXES captures both the expectations and the actual experiences of the user, one can measure the gap between the metrics and compare them, therefore providing a method to understand the user experience [49].

In the pre-survey (the first part of SUXES), participants evaluated their expectations based on an introductory video of the VR system shown to them. The post-survey had five sections with the statements answered on a 7-point Likert scale, where 1 = strongly disagree and 7 = strongly agree. The survey evaluated the participants’ actual experiences with the system (the second part of SUXES), views of the DocPanel tool, the perception of collaborative review sessions in VR, and perception of VR technology in general. In this paper, we use data from four sections of the surveys (both parts of the SUXES, the DocPanel tool, and collaborative review); data from the fifth section (perception of VR technology in general) has been published in another study.

After each user test session was completed, the main facilitator conducted a semi-structured group interview. Discussion revolved around topics such as general feelings and attitudes towards the tested system, participants’ evaluation of the system as a review and collaboration tool, and assessment of the current features and tools implemented in the environment. Participants were also asked what type of additional features or tools they would have liked to have been in the VR environment.

Qualitative data were collected during and after the test. During the test, observers noted down statements by the participants as they were thinking aloud and discussing with each other. After the session, further data were collected during the interview and noted down by the facilitator and observers. The data were analyzed with thematic analysis. Due to the small size of the test group, the statistical results are indicative only, but as domain experts are the real experts with their own tasks, the expert evaluation carries much weight in evaluating the usefulness and benefits of the environment for industrial maintenance tasks.

4. Results

Overall, experts left positive evaluations of collaborative reviews of technical documentation in COVE-VR. Figure 7 shows how experts perceive the value of collaborative reviews in VR by visualizing the division of answers via minimum, maximum and median of the answers for each statement. All experts agreed that review sessions in VR would positively affect the company’s overall performance, would accelerate the project span, and advance the knowledge transfer between the departments. Furthermore, eight experts agreed that collaborative review sessions in VR would help to identify more design errors and that a VR review session is more efficient than reviews via traditional conferencing tools. Finally, all experts agreed that review sessions in VR should be integrated to the company’s working practices.

The results of the SUXES survey, which compares the expectations and experiences with the VR system, are shown in Figure 8.

The survey results show that for most of the statements, the expectations of subject matter experts were met. They also demonstrate the overall positive evaluation of the system’s usability. Most of the experts found the system to be fast, pleasant, clear, easy to learn and natural to use, with a median at 5. In addition, all the participants found it to be useful. However, for about half of the experts, using the application was not effortless. The decrease between expectations and experiences happens in two statements—the experts expected the application to function less error-freely than experienced. In addition, less enthusiasm was demonstrated towards using the application in the future after experiencing it; however, no expert showed a negative attitude to this statement.

Figure 9 demonstrates how experts evaluated the DocPanel tool. The tool was found to be useful for the review processes and easy to use. In addition, experts believed that the tool would positively affect the collaborative review process and make it easier, faster, and more efficient. The results also show that the tools should be further advanced in terms of design and interactions.

The usefulness and benefits of VR technology to facilitate the collaborative review of technical documentation and the virtual tools were discussed in a semi-structured group interview. Despite the main focus of the interview being the DocPanel tool and the feasibility of the review process in VR, the participants were very engaged and gave many comments and improvement ideas on the other tools and the multiuser collaboration in general.

The concept of the DocPanel tool was evaluated as very useful by experts participating in the user testing. When reviewing technical instructions, one must have access to the actual document files; therefore, the instructions must be available in the VR. Participants were able to use the DocPanel tool and review the instructions in it while checking the components in VR. However, participants suggested several functionalities and improvements to the DocPanel tool that would enhance the review process in VR. Firstly, better navigation features would be needed. The DocPanel tool had basic next and back functionalities, but all participants agreed that a navigation pane or table of contents would be needed to obtain a comprehensive view of the instructions and to easily navigate to different parts of the instructions. With the next and back buttons, you can move inside one task, but navigating to a completely different part of the instruction is very cumbersome and laborious with them. The DocPanel had page numbers, but some participants commented that a progress bar would be a more suitable indicator of the progress made while reviewing a task. Secondly, all documentation experts commented on the need for markup or annotation tools for the DocPanel. The TextBox tool was used to take the notes, but as one could not attach a note to a specific page or a task in the DocPanel, it was thought of as quite clumsy. Users commented that attaching notes in the same way as with the commenting features in Adobe Acrobat would be a good addition to the DocPanel tool. Thirdly, users liked the idea of a floating window that you can move freely in the virtual environment. However, some users commented that the window was too small, and they would like it to be resizable so that you can freely decide what size suits you the best.

The participants discussed multiuser VR collaboration in length and agreed that it enhances both the documentation review process and the risk assessment process when compared to the current practices. The participants noticed an increased level of social presence and concentration on the task. Despite being physically located in different parts of the country or the world, the participants noted that VR would give a sense of being in the same room. One participant commented the following: “This is much more visual than the current process. You are forced to participate; you can’t read emails and so forth at the same time, but you have to concentrate on the task at hand.” The participants also suggested that desktop-based access to VR would be beneficial, calling people participating this way silent members or observers. The desktop participants would be then able to follow the review process in VR and also possibly take notes. One documentation expert suggested that an observer could be the one taking notes in the instructions outside of the VR in, for example, the PDF file. The VR participants would be then able to concentrate on reading and reviewing the instructions in the DocPanel tool, and already existing tools would be used to annotate and mark up the file by an observer. This type of hybrid setup would offer an easy adoption of VR, as good commenting tools already exist. However, it would require that an extra person is always available as an observer taking notes, which might prove problematic resourcing-wise. One documentation expert noted that if the session was recorded, they could watch it afterwards and make the needed changes in the instructions while watching the recording.

All the test participants agreed that even though COVE-VR would be useful for documentation review, you cannot review very long instructions in it but need to take breaks in between. Reviewing the whole instructions (e.g., the overview of the whole maintenance of a certain component) in VR would take quite some time with frequent breaks. Furthermore, many participants commented that documentation review in VR would mostly benefit the early draft reviews and entirely new tasks where you concentrate more on a specific task.

The participants agreed that risk assessments in VR would enhance the current process where the equipment to be risk assessed is not always available or accessible. They noted that it would be especially good for early risk assessments when the actual physical prototypes rarely exist. However, from a risk assessment point of view, the whole equipment needs to be modelled in VR in a way that it can be interacted with. In our tests, only certain components were modelled in such a way, and the risk assessment experts commented that you have to be able to interact with the full model or then have a blank virtual room with just the component you are reviewing in it. The risk assessment process takes the surroundings and environment into account, and the risk assessment for the method for replacing a component, for example, is seldom carried out on its own but rather reviewed in the context. Risk assessment experts also commented that haptic gloves and motion feedback would enhance the user experience, as you could also feel the objects you are touching. They also discussed the importance of importing standard maintenance tools, such as screwdrivers and wrenches, into the VR environment because the use of the tools is also considered in the risk assessment. The risk assessment experts also commented on the use of personal protective equipment and how it would be important to be able to model that in VR.

All the test participants commented on the need of a pointer tool to point out objects to others. In addition to the DocPanel navigation improvements, the pointer was the most requested enhancement proposal from the participants regardless of their role in the tests. One participant started using the measure tool as a pointer, placing it on objects he was talking about and stated the following: “Are you others able to see where I am pointing with this?” This further indicates that there is a great need for a pointer tool, and it would considerably enhance the collaboration in a multiuser VR environment. One test participant suggested that color-coded pointers would make it easy for everybody to recognize who is showing something. In addition, maintenance method developers asked for arrow and freeform drawing tools, as they would make it easier to explain details to others. Some participants also noted that a magnifying glass would be good so that details could be enlarged.

The participants enjoyed the multiuser collaboration in VR. They said that the avatars made it evident that they were not alone at the virtual equipment even though not everybody talked at the same time. However, several participants noted that realistic avatars with real faces would be good and would further enhance the collaboration and feeling of being in the same space, stating the following: “Avatars with real faces would be great, you would recognize people.” Some suggested that the Office365 picture of the persons could be used as the avatar as that is something they are used to viewing and would recognize immediately. Avatar heads used in COVE-VR were also viewed as too large and smaller ones would be good as the current heads get in the way of seeing things, especially in a cramped space with many concurrent users. Finally, the participants noted that “the VR is not a replacement for real equipment but a good addition”.

5. Discussion

This article presented the results of an expert case study on enhancing maintenance documentation review and risk assessment processes with the use of collaborative virtual reality. The study addressed the actual challenges in the industry, where access to physical equipment is limited or non-existent and experts work in different locations and are, many times, unable to meet face-to-face. Since the beginning of the 1990’s, many academic and industrial studies have demonstrated the value of VR for industrial operations in various fields [7,16,25,50]. However, even though the use of VR has been promoted in industrial companies, its main application areas in companies are still training and design reviews. Our study demonstrates that the use of VR can also enhance other research and development related processes in industrial companies. Previously, the cost of the hardware was noted as the greatest obstacle for VR adoption in companies [9], but as prices have come down considerably during the past few years, this is not a major issue any more and companies are investing more in VR and related equipment. Furthermore, for companies where VR technology has been already adopted, e.g., for training purposes, the integration of other processes and use cases for VR would be fairly easy to achieve. Exploring all the possible potential VR scenarios based on existing hardware would also boost the adoption of industry 4.0 interventions.

5.1. Benefits of VR to Collaboration and Inclusiveness

Previous studies reported that VR enhances communication and collaboration activities [23,26,51]. Accordingly, our study demonstrates that the greatest advantage of virtual reality for the maintenance documentation review and risk assessment processes is its positive effect on the collaboration of the team working together towards a common goal. Instead of people working independently and alone at their desktops or joining conference calls, VR offers them a collaboration platform where they have, despite of their physical location, a sense of being together in the same room [21,22,47,52]. Not only does VR enhance the current collaboration process by offering virtual access to equipment that is not available [4], it also promotes inclusiveness, as additional team members from other countries can easily join documentation review and risk assessment sessions from their own locations. The benefits of multiuser VR are evident when comparing it to the current practice of reviewing and commenting technical instructions (in PDF files or by attending conference calls in tools such as MS Teams), which are not thought of as very collaborative. Furthermore, when comparing to physically being present in the same meeting room, remote participation through multiuser VR enables diverse experts from other countries to engage without a need to travel and physically attend meetings. This is both a clear benefit for globally operating companies and their employees from both a cost and sustainability point of view. VR also provides more equal opportunities globally and facilitates viewpoints from globally scattered team members, benefiting both the multi-national company and its employees. Lifelike, realistic avatars would further improve the sense of togetherness and working as a team, as people would be easily recognized in VR [53].

5.2. Benefits of Collaborative VR to Documentation Review and Risk Assesment

The results of our user testing demonstrate that the concept of documentation review and risk assessment in VR was rated positively by the participants. Our concept was tested with the COVE-VR platform, but any collaborative VR environment with similar tools would offer an efficient platform for maintenance documentation review and risk assessment processes. The DocPanel tool offers the ability to test maintenance methods and concurrently review the technical instructions, even when there is no physical equipment available. In comparison to working independently with files on a laptop, collaborative VR offers the ability to show how a task is performed, to point out components, and to demonstrate their functionalities. It also introduces an enhanced sense of being together and working as a team. The user test participants noted that the combination of people and departments in our tests was good, but clear roles are needed so that everybody knows what to do. For example, before a review session starts, it must be defined who is responsible for operating the DocPanel and leading the documentation review and who takes notes of any needed changes.

Spatial understanding is essential for many industrial processes [54]. For example, in maintenance method development, it is important to understand whether there is enough space to carry out the maintenance task. The sense of scale is easily lost when looking at the 3D model from computer screen, which can lead to maintenance methods that are impossible to perform. The related maintenance instructions are then impossible to follow, which can then both frustrate the users and cause safety issues when the users invent their own way of performing the task. These kinds of mistakes are avoided with the 1:1 scale in VR, as VR creates a sense of spatial understanding.

5.3. Limitations of Collaborative VR in Documetation Review and Risk Assesment

Some limitations still exist in fully using VR for maintenance documentation review and risk assessment processes. Most of the user test participants noted that reviews in VR would be good for early drafts and early risk assessments. However, the 3D model might not be always ready and available in very early phases of product development. Further focus needs to be given, therefore, to integrating the early creation of 3D models to the product development process. Additionally, as the 3D model is often updated during the product development cycles, it would also be essential to easily update the VR model when there are changes in the 3D model. Additionally, it would be beneficial if the VR environment would be able to indicate the changes made in the 3D model so that recent changes can be easily noticed.

The quality of immersion and sense of presence improve the ability to identify risks. The modelling of tools and animating the movement of objects proposed in the results of this study agree well with other studies [40]. One problem for risk assessment in VR is that the environment is typically ’clean’, with no odors, no noise, no temperatures, or equivalent. Hazard identification is based only on the visual observation of environment [41]. Therefore, people performing risk assessment need to be aware of and competent enough to identify hidden hazards.

To improve the situation, we propose the following to enhance VR hazard identification: objects must have hazard-related metadata attached to them. This data can be made visible as an additional visualization layer that can be switched on and off. For example, objects connected to voltage sources could have a blue aura or shimmering, objects with chemical hazards a yellow aura, and hot objects a red aura. Different visualization, or audio feedback, if available, could be given similarly to any hazard, be it of mechanical origin, irradiation, pressure and so on.

Even though the concept of the DocPanel was rated positively, its implementation had its limitations. For the DocPanel to be an efficient tool, enhancements and additions would be needed especially in navigation and annotation tools. As people are used to the current navigation and commenting functionalities of common office tools, such as Adobe Acrobat and MS Word, replicating those in DocPanel would lower the learning curve for the users of COVE-VR. From a multiuser collaboration point of view, a pointer tool would be essential. Maintenance method developers, documentation experts and risk assessment experts discuss details when reviewing instructions and assessing risks, and many times need to point out a small detail. In real life, with access to real equipment, this would be carried out with a finger, and all the participants looked for a way of pointing a detail or component to others in VR. In addition, drawing tools would further enhance the collaboration features of COVE-VR. The development of the DocPanel and the related tools and their usefulness to the processes described in this paper would offer an interesting further research area.

5.4. Limitations of This Study and Areas of Further Research

This study’s limitation is the focus on a single industrial company’s documentation and risk assessment processes. However, documentation review and risk assessment are universal processes in industrial companies on a general level, and even if the details of the process may vary from one company to another, the processes are still collaborative by nature. Studying other companies’ processes and the usefulness of VR to those processes would offer further insight into how generalizable the results of this study are to the fields of documentation review and risk assessment. Additionally, the potential enhancements to the VR environment and tools suggested by the experts in our user study would offer an interesting development and further research area for collaborative VR.

6. Conclusions

Even though virtual reality environments are already in active use in many industrial companies, their use has been mainly focused on training or design reviews. However, VR has much to offer to other functions and product development departments, especially in the case of globally operating companies and globally scattered teams.

This study explored the benefits of VR to maintenance documentation review and risk assessment processes. The concept of reviews in VR and the DocPanel tool were evaluated by an industrial company’s domain experts in user tests. Overall, our study indicates the potential of VR as a tool to enhance maintenance documentation review and risk assessment processes. Even though the focus of this study was on a single industrial company’s documentation and risk assessment processes, the processes are universal processes used in other industrial companies as well. Therefore, the results are largely generalizable to other industrial companies and their processes. We used the COVE-VR platform in our study, but our any collaborative VR environment with similar tools would offer an efficient platform for maintenance documentation review and risk assessment.

The study demonstrates that VR had a positive effect on the collaboration of the cross-organizational team working towards a common goal. In globally operating multi-national companies where experts work in different locations and are, many times, unable to meet face-to-face, VR offers a collaboration platform, strengthens the sense of being part of a team, and promotes inclusiveness. It also gives virtual access to equipment in cases where the physical prototype does not exist or is inaccessible to the members of the team. VR also strengthens spatial understanding, and, therefore, results in more accurate maintenance methods and related maintenance instructions. Even though reviews in VR were not viewed as a replacement for documentation review and risk assessment processes regarding real equipment, VR was rated a very useful alternative in cases where access to the physical equipment is limited or non-existent.

Author Contributions

Conceptualization, H.H., A.B., S.S. and M.T.; methodology, H.H., A.B. and S.S.; software, A.B., validation, J.H., S.S. and M.T.; formal analysis, H.H. and A.B.; investigation, H.H., A.B. and S.S.; resources, S.S. and M.T.; data curation, H.H. and A.B.; writing—original draft preparation, H.H., A.B., S.S. and J.L.; writing—review and editing, H.H., A.B., S.S., J.L., J.H. and M.T.; visualization, H.H. and A.B.; supervision, S.S. and M.T.; project administration, S.S. and M.T.; funding acquisition, S.S. and M.T. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Informed consent was obtained from all subjects involved in the study.

Data Availability Statement

Not applicable.

Acknowledgments

We would like to warmly thank the global employees of KONE for participating in the user testing and providing their valuable insight. We also thank John Mäkelä and Kimmo Ronkainen from Tampere University for their contribution to the software development of COVE-VR.

Conflicts of Interest

The authors declare no conflict of interest.

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Figures

Figure 1. Maintenance documentation process.

Figure 2. Screenshot of collaborative review of technical documentation in VR. A user is immersed in VR and looking at the avatars of two other users, complete with name labels for identification.

Figure 3. Task step visualized in the DocPanel Tool (left), the same step in the original PDF instruction (right).

Figure 4. User study setup, showing both the video streams in Teams and the setups of the rooms and locations.

Figure 5. User tests in progress. Laptop streaming the participant’s VR view can be observed in the background. An USB camera is connected to the laptop in the background, streaming the physical space of the room to Teams.

Figure 6. User test participant and VR view displayed in a monitor.

Figure 7. The results of collaboration-related statements, answered on a seven-point Likert scale.

Figure 8. Expectations and experiences with VR platform. A seven-point Likert scale was used.

Figure 9. Evaluation of the DocPanel tool for collaborative review in VR. A seven-point Likert scale was used.

References

1. KONE Corporation—Improving the Flow of Urban Life. Available online: https://www.kone.com/en/ (accessed on 20 June 2022).

2. Hackos, J.T. Managing Your Documentation Projects; Wiley: New York, NY, USA, 1994.

3. Jayaprakash, S. Technical Writing; Himalaya: Mumbai, India, 2008.

4. Stock, I.; Weber, M.; Steinmeier, E. Metadata based authoring for technical documentation. Proceedings of the 23rd Annual International Conference on Design of Communication Documenting & Designing for Pervasive Information—SIGDOC ’05; Coventry, UK, 21–23 September 2005; ACM Press: Coventry, UK, 2005; 60.

5. Wolfartsberger, J. Analyzing the Potential of Virtual Reality for Engineering Design Review. Autom. Constr.; 2019; 104, pp. 27-37. [DOI: https://dx.doi.org/10.1016/j.autcon.2019.03.018]

6. Choi, S.S.; Jo, H.; Boehm, S.; Sang, D.N. ONESVIEW: An Integrated System for One-Stop Virtual Design Review. Concurr. Eng.; 2010; 18, pp. 75-91. [DOI: https://dx.doi.org/10.1177/1063293X10361624]

7. Guo, Z.; Zhou, D.; Zhou, Q.; Zhang, X.; Geng, J.; Zeng, S.; Lv, C.; Hao, A. Applications of Virtual Reality in Maintenance during the Industrial Product Lifecycle: A Systematic Review. J. Manuf. Syst.; 2020; 56, pp. 525-538. [DOI: https://dx.doi.org/10.1016/j.jmsy.2020.07.007]

8. Narasimha, S.; Dixon, E.; Bertrand, J.W.; Chalil Madathil, K. An Empirical Study to Investigate the Efficacy of Collaborative Immersive Virtual Reality Systems for Designing Information Architecture of Software Systems. Appl. Ergon.; 2019; 80, pp. 175-186. [DOI: https://dx.doi.org/10.1016/j.apergo.2019.05.009]

9. Zaker, R.; Coloma, E. Virtual Reality-Integrated Workflow in BIM-Enabled Projects Collaboration and Design Review: A Case Study. Vis. Eng.; 2018; 6, 4. [DOI: https://dx.doi.org/10.1186/s40327-018-0065-6]

10. Murray, N.; Fernando, T.; Aouad, G. A Virtual Environment for the Design and Simulated Construction of Prefabricated Buildings. Virtual Real.; 2003; 6, pp. 244-256. [DOI: https://dx.doi.org/10.1007/s10055-003-0107-8]

11. Ben Hadj, R.; Trigui, M.; Aifaoui, N. Toward an Integrated CAD Assembly Sequence Planning Solution. Proc. Inst. Mech. Eng. Part C J. Mech. Eng. Sci.; 2015; 229, pp. 2987-3001. [DOI: https://dx.doi.org/10.1177/0954406214564412]

12. Schwarz, S.; Regal, G.; Kempf, M.; Schatz, R. Learning success in immersive virtual reality training environments: Practical evidence from automotive assembly. Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society; Tallinn, Estonia, 25 October 2020; pp. 1-11.

13. Andaluz, V.H.; Sánchez, J.S.; Sánchez, C.R.; Quevedo, W.X.; Varela, J.; Morales, J.L.; Cuzco, G. Multi-user industrial training and education environment. Augmented Reality, Virtual Reality, and Computer Graphics; De Paolis, L.T.; Bourdot, P. Lecture Notes in Computer Science Springer International Publishing: Cham, Switzerland, 2018; Volume 10851, pp. 533-546. ISBN 978-3-319-95281-9

14. Burova, A.; Mäkelä, J.; Hakulinen, J.; Keskinen, T.; Heinonen, H.; Siltanen, S.; Turunen, M. Utilizing VR and Gaze Tracking to Develop AR Solutions for Industrial Maintenance. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems; Honolulu, HI, USA, 21 April 2020; pp. 1-13.

15. Pfeiffer-Leßmann, N.; Pfeiffer, T. ExProtoVAR: A Lightweight Tool for Experience-Focused Prototyping of Augmented Reality Applications Using Virtual Reality. Proceedings of the HCI International 2018—Posters’ Extended Abstracts; Las Vegas, NV, USA, 15–20 July 2018; Stephanidis, C. Springer International Publishing: Cham, Switzerland, 2018; pp. 311-318.

16. Berg, L.P.; Vance, J.M. An Industry Case Study: Investigating Early Design Decision Making in Virtual Reality. J. Comput. Inf. Sci. Eng.; 2016; 17, 011001. [DOI: https://dx.doi.org/10.1115/1.4034267]

17. Wolfartsberger, J.; Zenisek, J.; Sievi, C.; Silmbroth, M. A Virtual Reality Supported 3D Environment for Engineering Design Review. Proceedings of the 2017 23rd International Conference on Virtual System & Multimedia (VSMM); Dublin, Ireland, 31 October–4 November 2017; pp. 1-8.

18. Berni, A.; Borgianni, Y. Applications of Virtual Reality in Engineering and Product Design: Why, What, How, When and Where. Electronics; 2020; 9, 1064. [DOI: https://dx.doi.org/10.3390/electronics9071064]

19. De Silva, R.K.J.; Rupasinghe, T.D.; Apeagyei, P. A Collaborative Apparel New Product Development Process Model Using Virtual Reality and Augmented Reality Technologies as Enablers. Int. J. Fash. Des. Technol. Educ.; 2019; 12, pp. 1-11. [DOI: https://dx.doi.org/10.1080/17543266.2018.1462858]

20. Churchill, E.F.; Snowdon, D. Collaborative Virtual Environments: An Introductory Review of Issues and Systems. Virtual Real.; 1998; 3, pp. 3-15. [DOI: https://dx.doi.org/10.1007/BF01409793]

21. Bailey, J.O.; Bailenson, J.N. Virtual Reality and Collaboration; Oxford University Press: Oxford, UK, 2014.

22. Du, J.; Shi, Y.; Zou, Z.; Zhao, D. CoVR: Cloud-Based Multiuser Virtual Reality Headset System for Project Communication of Remote Users. J. Constr. Eng. Manag.; 2018; 144, 04017109. [DOI: https://dx.doi.org/10.1061/(ASCE)CO.1943-7862.0001426]

23. Abbas, A.; Choi, M.; Seo, J.; Cha, S.H.; Li, H. Effectiveness of Immersive Virtual Reality-Based Communication for Construction Projects. KSCE J. Civ. Eng.; 2019; 23, pp. 4972-4983. [DOI: https://dx.doi.org/10.1007/s12205-019-0898-0]

24. Lepola, A.; Hannu, K.; Jalo, H.; Torro, O. Collaborative Virtual Reality as an Adaptable Boundary Object in the Design Phase of Facility Life Cycle. Proceedings of the 12th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management; Budapest, Hungary, 2–4 November 2022; SCITEPRESS-Science and Technology Publications: Setúbal, Portugal, 2020; Volume 3, pp. 63-75.

25. Tea, S.; Panuwatwanich, K.; Ruthankoon, R.; Kaewmoracharoen, M. Multiuser Immersive Virtual Reality Application for Real-Time Remote Collaboration to Enhance Design Review Process in the Social Distancing Era. J. Eng. Des. Technol.; 2022; 20, pp. 281-298. [DOI: https://dx.doi.org/10.1108/JEDT-12-2020-0500]

26. Wolfartsberger, J.; Zenisek, J.; Wild, N. Supporting Teamwork in Industrial Virtual Reality Applications. Procedia Manuf.; 2020; 42, pp. 2-7. [DOI: https://dx.doi.org/10.1016/j.promfg.2020.02.016]

27. Becerril Palma, P. Enhancing Lean and Agile Ways of Working Using an Asymmetric Multi-User Virtual Reality Setup. Master’s Thesis; Tampere University: Tampere, Finland, 2021.

28. Gamlin, A.; Breedon, P.; Medjdoub, B. Immersive virtual reality deployment in a lean manufacturing environment. Proceedings of the 2014 International Conference on Interactive Technologies and Games; Nottingham, UK, 16–17 October 2014; pp. 51-58.

29. Nevo, S.; Chengalur-Smith, I. Enhancing the Performance of Software Development Virtual Teams through the Use of Agile Methods: A Pilot Study. Proceedings of the 2011 44th Hawaii International Conference on System Sciences; Kauai, HI, USA, 4–7 January 2011; pp. 1-10.

30. Schäfer, A.; Reis, G.; Stricker, D. A Survey on Synchronous Augmented, Virtual and Mixed Reality Remote Collaboration Systems. ArXiv; 2021; arXiv: 210205998[DOI: https://dx.doi.org/10.1145/3533376]

31. Siltanen, S.; Heinonen, H.; Burova, A.; Turunen, M. Reimagining Innovation for Remote, Distributed Teams with Collaborative Virtual Reality. Proceedings of the ISPIM “Reconnect, Rediscover, Reimagine”; Valencia, Spain, 29 November–1 December 2021; pp. 1-14.

32. Defining Technical Communication. Available online: https://www.stc.org/about-stc/defining-technical-communication/ (accessed on 15 May 2022).

33. Scholz, J. Developing Technical Documentation the Smart Way. Technol. Commun.; 1994; 41, pp. 94-99.

34. Pringle, A.S.; O’Keefe, S. Technical Writing 101: A Real-World Guide to Planning and Writing Technical Documentation; Scriptorium Press: Research Triangle Park, NC, USA, 2000; ISBN 978-0-9704733-0-1

35. Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on Machinery, and Amending Directive 95/16/EC (Recast). Off. J. Eur. Union; 2006; 49, L157.

36. Directive 2014/33/EU—Lifts and Safety Components for Lifts (Recast) | Safety and Health at Work EU-OSHA. Available online: https://osha.europa.eu/en/legislation/directive/directive-201433eu-lifts-and-safety-components-lifts-recast (accessed on 18 May 2022).

37. EN 13015:2001+A1:2008—Maintenance for Lifts and Escalators—Rules for Maintenance Instructions. Available online: https://standards.iteh.ai/catalog/standards/cen/07ca9a23-173d-445a-839e-65a429305ab6/en-13015-2001a1-2008 (accessed on 18 May 2022).

38. 14:00–17:00 ISO 12100:2010. Available online: https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/05/15/51528.html (accessed on 18 May 2022).

39. 14:00–17:00 ISO 14798:2009. Available online: https://www.iso.org/cms/render/live/en/sites/isoorg/contents/data/standard/04/60/46048.html (accessed on 18 May 2022).

40. Puschmann, P.; Horlitz, T.; Wittstock, V.; Schütz, A. Risk Analysis (Assessment) Using Virtual Reality Technology—Effects of Subjective Experience: An Experimental Study. Procedia CIRP; 2016; 50, pp. 490-495. [DOI: https://dx.doi.org/10.1016/j.procir.2016.04.115]

41. Slezak, R.J.; Keren, N.; Finseth, T. Virtual Reality Application for Enhancing Risk Assessment Skills. Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC); Orlando, FL, USA, 26 November 2018; Paper No. 18280 11p.

42. Da Silva, A.G.; Mendes Gomes, M.V.; Winkler, I. Virtual Reality and Digital Human Modeling for Ergonomic Assessment in Industrial Product Development: A Patent and Literature Review. Appl. Sci.; 2022; 12, 1084. [DOI: https://dx.doi.org/10.3390/app12031084]

43. Barkokebas, R.; Ritter, C.; Sirbu, V.; Li, X.; Al-Hussein, M. Application of Virtual Reality in Task Training in the Construction Manufacturing Industry. ISARC Proc.; 2019; 36, pp. 796-803.

44. De-Juan-Ripoll, C.; Soler-Domínguez, J.L.; Guixeres, J.; Contero, M.; Álvarez Gutiérrez, N.; Alcañiz, M. Virtual Reality as a New Approach for Risk Taking Assessment. Front. Psychol.; 2018; 9, 2532. [DOI: https://dx.doi.org/10.3389/fpsyg.2018.02532]

45. Leder, J.; Horlitz, T.; Puschmann, P.; Wittstock, V.; Schütz, A. Comparing Immersive Virtual Reality and Powerpoint as Methods for Delivering Safety Training: Impacts on Risk Perception, Learning, and Decision Making. Saf. Sci.; 2019; 111, pp. 271-286. [DOI: https://dx.doi.org/10.1016/j.ssci.2018.07.021]

46. Burova, A.; Heinonen, H.; Becerril Palma, P.; Keskinen, T.; Hakulinen, J.; Opas, V.; Mäkelä, J.; Ronkainen, K.; Siltanen, S.; Raisamo, R. Toward Efficient Academia-Industry Collaboration: A Case Study of Joint VR System Development. Proceedings of the Academic Mindtrek 2021 (Mindtrek ’21); Tampere, Finland, 1–3 June 2021; ACM: New York, NY, USA, 2021; pp. 176-185.

47. Burova, A.; Palma, P.B.; Truong, P.; Mäkelä, J.; Heinonen, H.; Hakulinen, J.; Ronkainen, K.; Raisamo, R.; Turunen, M.; Siltanen, S. Distributed Asymmetric Virtual Reality in Industrial Context: Enhancing the Collaboration of Geographically Dispersed Teams in the Pipeline of Maintenance Method Development and Technical Documentation Creation. Appl. Sci.; 2022; 12, 3728. [DOI: https://dx.doi.org/10.3390/app12083728]

48. Burova, A.; Mäkelä, J.; Heinonen, H.; Palma, P.B.; Hakulinen, J.; Opas, V.; Siltanen, S.; Raisamo, R.; Turunen, M. Asynchronous Industrial Collaboration: How Virtual Reality and Virtual Tools Aid the Process of Maintenance Method Development and Documentation Creation. Comput. Ind.; 2022; 140, 103663. [DOI: https://dx.doi.org/10.1016/j.compind.2022.103663]

49. Turunen, M.; Hakulinen, J.; Melto, A.; Heimonen, T.; Keskinen, T.; Hella, J. SUXES—User Experience Evaluation Method for Spoken and Multimodal Interaction. Proceedings of the Tenth Annual Conference of the International Speech Communication Association; Brighton, UK, 6–10 September 2009.

50. Clergeaud, D.; Roo, J.S.; Hachet, M.; Guitton, P. Towards Seamless Interaction between Physical and Virtual Locations for Asymmetric Collaboration. Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology; Gothenburg, Sweden, 8 November 2017; pp. 1-4.

51. Bleakley, A.; Wade, V.; Cowan, B.R. Finally a Case for Collaborative VR?: The Need to Design for Remote Multi-Party Conversations. Proceedings of the 2nd Conference on Conversational User Interfaces; Bilbao, Spain, 22 July 2020; pp. 1-3.

52. Ayala García, A.; Galván Bobadilla, I.; Arroyo Figueroa, G.; Pérez Ramírez, M.; Muñoz Román, J. Virtual Reality Training System for Maintenance and Operation of High-Voltage Overhead Power Lines. Virtual Real.; 2016; 20, pp. 27-40. [DOI: https://dx.doi.org/10.1007/s10055-015-0280-6]

53. Pakanen, M.; Alavesa, P.; van Berkel, N.; Koskela, T.; Ojala, T. “Nice to See You Virtually”: Thoughtful Design and Evaluation of Virtual Avatar of the Other User in AR and VR Based Telexistence Systems. Entertain. Comput.; 2022; 40, 100457. [DOI: https://dx.doi.org/10.1016/j.entcom.2021.100457]

54. Piumsomboon, T.; Lee, Y.; Lee, G.; Billinghurst, M. CoVAR: A Collaborative Virtual and Augmented Reality System for Remote Collaboration. Proceedings of the SIGGRAPH Asia 2017 Emerging Technologies; Bangkok, Thailand, 27 November 2017; pp. 1-2.