Content area
As computing power and ubiquity increase, so do the opportunities of enhancing every day activities using technologies such as augmented (AR) or virtual reality (VR). While the former has seen extensive use on assembly/service lines, the latter has broader usage domains such as entertainment, remote meetings, remote medical assistance, social networking and many other. Preparing future VR developers usually requires expensive hardware and software as well as prebuilt environments. This paper proposes a novel method of teaching VR to students using A-FRAME, an open-source framework that promises to make VR as easy to create as a web page. The framework extends the Hyper-Text Markup Language (HTML) and Javascript, that together form the basis of all web-pages, by adding VR specific objects and functions thus allowing VR to be integrated into normal web pages. The paper presents the traditional and eLearning methods used to teach students how to assemble a virtual world, add real-life objects (a TV, a DSLR camera and a smart phone) and a user-interface, all with the end objective of building a virtual electronics shop. The virtual shop has all the features of its real counterpart such as commercials on TV-screens, text details as well as a 360-degree view of each product. Each separate task was taught as a challenge that involved different software tools and methods. The template application can be tested online on any computer or mobile phone (even using Google cardboard-like headsets) or dedicated VR headsets. All associated material and source code is available on Github.
Abstract: As computing power and ubiquity increase, so do the opportunities of enhancing every day activities using technologies such as augmented (AR) or virtual reality (VR). While the former has seen extensive use on assembly/service lines, the latter has broader usage domains such as entertainment, remote meetings, remote medical assistance, social networking and many other. Preparing future VR developers usually requires expensive hardware and software as well as prebuilt environments. This paper proposes a novel method of teaching VR to students using A-FRAME, an open-source framework that promises to make VR as easy to create as a web page. The framework extends the Hyper-Text Markup Language (HTML) and Javascript, that together form the basis of all web-pages, by adding VR specific objects and functions thus allowing VR to be integrated into normal web pages. The paper presents the traditional and eLearning methods used to teach students how to assemble a virtual world, add real-life objects (a TV, a DSLR camera and a smart phone) and a user-interface, all with the end objective of building a virtual electronics shop. The virtual shop has all the features of its real counterpart such as commercials on TV-screens, text details as well as a 360-degree view of each product. Each separate task was taught as a challenge that involved different software tools and methods. The template application can be tested online on any computer or mobile phone (even using Google cardboard-like headsets) or dedicated VR headsets. All associated material and source code is available on Github.
Keywords: virtual reality; a-frame; academic teaching.
I.INTRODUCTION
As technology advances so rapidly, computing devices are continuously integrated into everyday life. Since the introduction of 3D graphics hardware and software and their mainstream adoption by the gaming industry, there have been numerous attempts to enhance the current user experience that is tied to 3D images that are projected onto a 2D screen. A relatively recent technological branch is that of augmented reality (AR) and virtual reality (VR). These two differ significantly in that the former attempts to project 3D objects onto an otherwise real image/video, while the latter creates an entire world that can be navigated by the user.
VR and AR are expected to lead the new wave of digital technologies, in terms of economic and social impact. It is estimated that the economic impact of VR/AR technologies by 2020 to be 15.6 billion USD, while the consumer VR software and hardware market size worldwide will reach 14.5 billion USD in 2021 [10].
Creating a virtual world usually requires some knowledge of 3D graphics, 3D models and experience with a specific target device and its accompanying application programing interface (API). Any AR/VR developer must handle all of these tasks in order to create a usable application. In an attempt to simplify the development process, many other information technology areas have produced specialized frameworks such as for the Web, smartphones and even embedded devices. Continuing this trend, a WebVR framework named A-Frame was introduced in 2015.
This paper proposes an introduction to the world of VR using open-source software such as web frameworks (A-Frame), 3D editing suites (Blender), collaborative online code editors (Glitch) and eLearning platforms (Moodle). The educational objective of this integrated approach is to demonstrate the step by step construction of a virtual electronics shop complete with products that can be viewed, selected and can even show video content on-screen. This scenario is used in our current teaching for 50 last year students of the Polyethnic University of Bucharest, taking a course in Robotics and Virtual Reality.
After a short literature review, this paper continues with a description of the A-Frame WebVR framework, that is followed by a pedagogical case study. The construction of the aforementioned virtual shop is broken into four distinct stages that correspond to the four subsections of the case study. The case study starts with a basic virtual scene, adds 3D models, creates a graphical user interface (GUI) and finally, modifies one of the models such that it can display movies on its screen. The paper ends with conclusions and directions for further developments.
II.RELATED WORK
Using VR to support learning has already gained popularity among teachers at all levels and interests. A plethora of applications in various areas has been reported, such as remote physical training ([2]), geography learning ([7]), learning factories ([6]), or healthcare ([1]), to name only a few.
In the general eLearning context, the use of Learning Management Systems such as Moodle prevails as it allows for simple and effective course management as well as student evaluation and grading [11]. For particular academic domains such as engineering and computer science, the use of distrusted source code editing and automated deployment tools such as Glitch [12] can also improve student collaboration and code comprehension.
VR worlds are composed primarily of 3D models that are available under a free license or can be bought individually. Once a model file has been obtained, it can be modified using special 3D pipeline software such as Blender [5]. Reasons for editing a 3D model include adding animations, adding new model components or changing the aspect of the model by altering its material/texture.
There are already AR applications that allow users to visualize how a certain product might look in their home by rendering a 3D model on top of a live video stream from the smartphone camera [13].
III.A-FRAME, A WEBVR TECHNOLOGY
Traditionally, VR experiences required specialized hardware that was capable of rendering 3D scenes and detecting head movement. This approach, when coupled with desktop software with tight performance requirements, had a limiting impact on the portability and wide spread adoption and development of VR.
Due to major improvements in browser performance, particularly the introduction of HTML5 in 2008 and its canvas rendering [14], along with improvements in JavaScript client code execution, there was a clear prospect of enhancing the 2D paper-like webpage using rich multi-media content. One such enhancement was the introduction of WebGL, a 3D rendering API that was a JavaScript adaptation of the OpenGL ES 3 framework for use on Web applications [15]. Afterwards, the concept of WebVR appeared with the stated objective of bringing VR applications to the Web browser. The advantages of such a transition are the following [8]:
* Rapid access: there is no need to install software as the entire VR experience is running in the Web browser as a normal Web page;
* Easily reference Web pages/content/services
* Portability: HTML5 and WebGL are supported by browsers on all major desktop and mobile platforms;
* Simple deployment: the lack of installations means that a simple Web page reload is all that is needed to update an application.
A-Frame entered the scene in 2015 as a project of Mozilla, that could produce 3D VR scenes using an extension of the HTML markup language. A-Frame is based on an entity-component-system framework that allows for the creation of reusable components that can be grouped into entities for later use that, in turn, provides greater flexibility by avoiding long inheritance chains that are usually encountered in object-oriented programming [16].
Components can be defined using the JavaScript API while entities are defined in a declarative manner using HTML attributes, as shown in Figure 1. Each of the four components in this example (geometry, material, light, position) accepts a set of parameters that are separated by semicolons. In this example, a new cube entity is created with a brown color, is positioned (width, height, depth) at (0 4 -5) and emits a brown ambient light upon the scene. The entity can receive any number of features by adding new community developed components such as physics constraints or interactivity with the user [16].
IV.PEDAGOGICAL CASE STUDY - MY VIRTUAL SHOP
With the proliferation of e-commerce businesses, it will be a matter of time until they reach the virtual 3D realm using technologies such as VR. As specified earlier, AR applications do not require the creation of complete scenes but rather just the product model.
In this section, a case study on building a virtual shop is presented in a pedagogical manner. The intent is to highlight the simplicity and efficiency of the A-Frame environment as well as the use of modern online collaborative development environments as a complement to well established elearning platforms.
This section is composed of four increasingly difficult tasks that take the student from the equivalent "hello-world" VR application to a fully interactive scene. The source code for the entire case study is available on Github [3]. Students can either download this source code and run it locally or they can use an online service such as Glitch. An editable mirror of the Github repository is available on Glitch [4] and presents itself as an interactive webpage that contains a file tree on the left and a code editor. By clicking the right-side button labeled "Remix to Edit" the user can create an editable personal copy of the project, even without logging in. After making the necessary changes the user must click the button labeled "Show Live" in order to see test out the web application. Each task is accompanied by an HTML file that will be specified during the corresponding section.
The applications can be tested on computers as well as mobile devices as long as they use an up-to-date browser that supports WebVR and WebGL. By clicking the mask-like button on the bottom-right side of the scene, the user can activate full-screen mode on computers and cardboard-like VR mode on mobile devices.
4.1Creating a basic VR scene
This scene is one of the simplest that can be produced as it only makes use of geometric primitives such as cubes, cylinders, spheres and planes. This scene is also the first example shown on the homepage of A-Frame and is associated with the file named basic_scene.html.
From a pedagogical perspective, this scene is appropriate for an introduction to VR as it contains a minimal amount of code (15 lines) and the actual scene is constructed using only 6 lines of code, as shown in Figure 2.
The scene was defined by placing four entities within the a-scene system, part of the entity-component-system architecture discussed earlier. The effect is that all child entities are positioned relatively to their parent, in this case the global scene. As do normal entities, the system also accepts various parameters, in this case the background color.
Each of the four primitive entities has position, rotation and color components as well as specific dimension-related components such as width, height or radius. The shadow component enables the entities to cast and receive shadows. In the absence of manually defined light sources, A-Frame automatically introduces a white light entity.
The scene can be navigated by means of the keys W, A, S, D and the mouse, a configuration that is often employed in computer games. The user can also edit the scene graphically using a built-in scene editor that is started using the key combination CTRL+ALT+I.
4.2Adding 3D models
After creating a basic scene using primitive geometric entities, the next step towards creating a convincing VR world is adding real-life objects. This is achieved by loading 3D model files and placing them in the scene. This scene consists of a table and two products, an LCD TV and a DSLR camera and is stored in the file named hello_model.html.
This scene introduces the a-assets entity and gltf-model component (shown in Figure 3) that are specialized in resource loading and rendering of models respectively. This separation is necessary in order to optimize the resource loading/rendering process. The products are placed on the table using relative positioning by defining the product models as children of the table model.
From a pedagogical perspective, this task builds upon the first task by adding complex objects and texturing them such that they look real. The emphasis is not on the actual model creation but rather on describing the basic concepts that make 3D models possible. The Graphics Library Transmission Format (glTF) is an open standard of a generic 3D model format that is developed by the industry consortium Khronos Group [9]. The model files are composed of a text component for storing the model hierarchy, a binary component that stores the model geometry using vertices and faces and lastly, a set of images that are used as textures that cover the model and give it a natural look.
4.3Creating a Graphical User Interface
Most e-commerce sites offer details for each product such as make, model and specifications. These details are often visible only after opening a separate web page by clicking on a product from the list.
In a VR world, there is still a need for interactivity, but without specialized pointer hardware the user is limited to looking around (gaze) and moving. This problem was tackled in A-Frame using ray tracing algorithms that create virtual "laser" beams that can detect the objects that they pass through and return the first encounter. In A-Frame there are community-supported libraries that add a graphical user interface (GUI) that can be controlled by simply looking at a certain object for a fixed amount of time.
One such GUI library is used in this scene (available in heiio_gui.html) to add a cursor, a label, two buttons and to make the products selectable, as shown in Figure 4. Together, these interface features allow the user to select a product and afterwards chose to either buy the product or view its technical specifications by clicking on the corresponding button. This degree of interaction cannot be achieved using a purely declarative syntax (HTML) so JavaScript is used to respond to new events such as object selections.
The first step towards interactivity is adding a cursor (a-gui-cursor) in front of the camera. The cursor is a composite component that includes an animated graphic, in this case a dot, and a ray caster. The latter is responsible for generating a collision event for the first object that it encounters. For added flexibility, the ray caster component also allows one to whitelist certain objects that should be detected by specifying a list of valid (HTML) attributes. In this case, each of the products has the gui-interactable attribute.
The second step is the identification of individual products and the handling of click events. This double role is achieved simultaneously by defining a customized component as shown in Figure 5. The definition of the component is similar to that of a class in object-oriented languages in that it contains a list of attributes, the schema, and a constructor function, init, that is called during each instantiation of the component. The data member allows access to the user assigned values for each attribute defined in the schema. The last step of the constructor is to register an event handler for the click event. After this registration, the testProductAction() function will be called with the id that corresponds to the selected object. In this simple example, the function only sets the name of the selected product as the label above the buttons and also stores it in the product global variable.
The third and last step is the addition of GUI elements such as buttons and labels in the scene. For this purpose, the a-gui-flex-container and a-gui-button entities are used. The former is a container of other GUI entities while the latter defines a button that, when clicked, calls the JavaScript function testButtonAction('buy'). This in turn will open a new browser tab and navigate to an example Web page.
4.4Enhancing products
The currently employed e-commerce technology allows customers to view a list of products, details and reviews about a particular product and also pictures and in some cases rotatable views of the product. Instead, real electronics shops provide running products that display a demo on their screen in order to showcase device features such as viewing angle or color intensity.
In this section there is presented an approach of bridging this multimedia gap between real shops and their online counterparts by harnessing the power of VR. With that objective in mind, the TV model will be slightly modified in order to play a movie on its screen while the client is looking at it in VR. The soundtrack of the movie is also played. This scene is available in hello_tv_video. html.
Compared to the previous tasks, this one also requires basic knowledge of the building blocks of 3D graphics such as the vertex, edge, face, material and texture. Although they might seem complex, these relate closely to spatial (3D) geometry that is regularly taught in high school. The vertex is the equivalent of a point in space, the edge connects two vertices and the face is a group of three or four vertices forming either a triangle or a square, both of which can be directly rendered on screen by the graphics card. The material defines the light reflection/refraction properties of the object and also its color. A texture can be assigned to a material such that instead of a color, the material will map an image onto the given object.
These basic 3D graphics components are needed because the objective of this section is achieved by playing a video file and rendering each frame as a texture that is assigned solely to the TV screen and not its housing.
The first step is the actual segmentation of the TV screen from the rest of the model, seen in Figure 6b, using a 3D editing software such as Blender which is also open-source. Using its face selection mode, the user has to select all of the faces that make up the screen and join them into a mesh. After that, the screen mesh is given a human readable name such as "tv_screen".
The second step is adding the actual film into the Web page using the <video> tag that is available in HTML5 as shown in the top side of Figure 6a. This tag is usually used to play video files in a frame inside a normal Web page. Here it is only used as a data source and is hidden from view. Notice also that is has the unique id video.
The third step consists of the actual JavaScript code that projects frames from the video onto the TV screen texture. Firstly, the named mesh that was separated in the previous step is now searched for in the 3D model node tree using the traversen function. Secondly, the video tag is referenced using its unique identifier (video in this case) and used to create a special type of texture, the video texture, offered by the ThreeJS library. Thirdly, the TV screen mesh is assigned a newly created material that will display the video texture. After this last step, the movie is started in the video tag while the rendering is done onto the video texture.
V.CONCLUSIONS
The importance and impact of AR and VR technologies are increasingly significant in many domains. One might expect that the difficulty of designing products that use these technologies is correspondingly high but, as this paper shows, new technologies simplify this task greatly. Using WebVR technologies such as the A-Frame framework, beginner and expert developers alike can create useful VR worlds that are as accessible as the Web itself.
This paper also explored the challenges involved in introducing a novice to the world of VR applications through a step-by-step case study. The technical and pedagogical aspects of each separate stage of the project were discussed in detail.
Web services were also used in this paper as the project boilerplate by providing means of distributed storage, editing, viewing and collaboration over project files. Examples include the use of Github for storing the project files and modification history (commits) as well as the use of Glitch for online, collaborative editing and viewing of source code. These two services help improve student experimentation and collaboration in much the same way as Learning Management Systems such as Moodle can enhance the overall learning experience.
As for future developments, these are mainly focused on interactivity because currently the main method of interaction with the (A-Frame) VR world is by gazing on a particular element, as shown in this paper. Currently, there are commercial VR sets, such as the HTC Vive, that offer two hand-held wireless controllers, but the hardware is proprietary and cannot be easily adapted for scientific work. Thus, we are proposing an open hardware/software alternative that will be developed using commodity hardware and will facilitate the adding of new functions and VR interaction methods.
Reference Text and Citations
[1] Robyn Cant, Simon Cooper, Roland Sussex, and Fiona Bogossian. 2019. What's in a Name? Clarifying the Nomenclature of Virtual Simulation. Clinical Simulation in Nursing 27, (February 2019), 26-30. D0I:https://doi.org/10.1016/j.ecns.2018.11.003
[2] Xiaoming Chen, Zhibo Chen, Ye Li, Tianyu He, Junhui Hou, Sen Liu, and Ying He. 2019. ImmerTai: Immersive Motion Learning in VR Environments. Journal of Visual Communication and Image Representation 58, (January 2019), 416-427. DOI:https://doi.org/10.1016/j.jvcir.2018.n.039
[3] Andrei G. Florea and Cătălin Buiu. 2019. Github - Virtual Reality case study using A-Frame. Retrieved January 15, 2019 from https://github.com/andrei91ro/vr_case_study
[4] Andrei George Florea and Cătălin Buiu. Glitch - Virtual Reality case study using A-Frame. Retrieved January 15, 2019 from https://glitch.com/edit/# !/andrei91ro-vr-case-study
[5] Blender Foundation. About Blender. Retrieved from https://www.blender.org/about/
[6] Max Juraschek, Lennart Büth, Gerrit Posselt, and Christoph Herrmann. 2018. Mixed Reality in Learning Factories. Procedía Manufacturing 23, (January 2018), 153-158. DOI:https://doi.org/10.1016/j.promfg.2018.04.009
[7] Zhihan Lv, Xiaoming Li, and Wenbin Li. 2017. Virtual Reality Geographical Interactive Scene Semantics Research for Immersive Geography Learning. Neurocomputing 254, (September 2017), 71-78. DOI:https://doi.org/10.1016/j.neucom.2016.07.078
[8] Tony Parisi. 2015. Learning Virtual Reality. O'Reilly Media, Inc.
[9] 2013. glTF - Runtime 3D Asset Delivery. The Khronos Group. Retrieved January 16, 2019 from https://www.khronos.org/gltf/
[10] Virtual Reality (VR) - Statistics & Facts | Statista. Retrieved from https://www.statista.com/topics/2532/virtualreality-vr/
[11] AboutMoodle - MoodleDocs. Retrieved from https://docs.moodle.org/36/en/About_Moodle
[12] About Glitch. Retrieved from https://glitch.com/about/
[13] Augmented Reality for lifestyle ecommerce - Cimmerse. Retrieved January 15, 2019 from https://www.cimmerse. com/
[14] HTML 5. Retrieved January 15, 2019 from https://www.w3.org/TR/2008/WD-html5-20080122/
[15] WebGL 2.0 Specification. Retrieved January 15, 2019 from https://www.khronos.org/registry/webgl/specs/latest/2.0/
[16] Entity-Component-System. A-Frame. Retrieved January 15, 2019 from https://aframe.io/docs/0.8.0/introduction/entity-component-system.h1ml
Copyright "Carol I" National Defence University 2019