Application of User Experience and Design Thinking to the Construction of a Class Assistance System for Hearing- and Speech-Impaired People

Abstract

According to a study by the Department of Statistics, Ministry of Health and Welfare, Taiwan, a total of 15,145 people had a voice function impairment or speech function impairment, and a total of 122,533 people had a hearing impairment by the second quarter (Q2) of 2018. Hearing and speech are the most commonly used sensory functions for communication. Therefore, a person will suffer unimaginable difficulties with the loss of both functions during her/his life. The problems that a handicapped student might encounter during school are often neglected. The way a teacher teaches during a class and how a student interacts with others should be highlighted. The deficiency in assistive devices for hearing- and speech-impaired people is due to several factors. In fact, there is insufficient information and not enough researchers. The translation of language by foreign assistive devices is more rigorous; however, domestic resources for assistive devices for hearing- and speech-impaired people are limited. No relevant information on learning through assistive devices is available. The subject examined in this study was a hearing-impaired student in her junior year, who lost speech capability after an ear stroke. This study examined the problems that she encountered during her learning in a class with average students. After a literature review and the investigation of the case study, user experience and design thinking approaches were implemented for the development of an assistive device. A class learning assistive system, which is based on a simple innovative design and a highly flexible combination of elements, was created. After a series of experimental verifications and improvements, the results indicated that this learning assistive system could effectively enhance the research subject’s confidence and autonomy during learning in class.

1. Introduction

A hearing-impaired person lacks hearing capability, and a speech-impaired person lacks the speech capability; both losses lead to extreme inconvenience during daily life. For a student, there are more challenges to overcome during learning, such as communication between classmates, learning during class, acquirement of information, etc.

According to the official data for the second quarter of 2018 from the Department of Statistics, Ministry of Health and Welfare, Taiwan, the number of people with a speech impairment in Taiwan is about 15,145. The number of people with a hearing dysfunction also reaches 122,533. Hearing and speech are the most commonly used approaches for communication. In our daily life, people can recognize the orientation of a sound source by their ears. Therefore, hearing is an important sensory function to receive messages and to communicate. A hearing loss might be due to a congenital deficiency, a disease, or san accident. Li [1] proposed that hearing in a person with minor hearing loss can be improved by a hearing aid. For a person with severe hearing loss, speech training is also required in addition to wearing a hearing aid. Moreover, some hearing-impaired people might use sign language for communication. However, not every hearing-impaired person knows sign language, and therefore, there are limitations and difficulties for a hearing-impaired person to communicate or receive messages. Talking is the most direct and fastest way of communicating or expressing oneself [2]. Communication by voicing out can achieve mutual understanding or convey messages. According to a study by Changchun Deaf [3], the hearing organ is different from the speech organ, since the speech capability is not affected if the speech organ develops well. However, an inborn hearing problem could hamper speech learning due to the inability to hear voices. The problem of hearing loss is due to a person gradually forgetting how to voice, as his/her hearing is impaired. He/she lacks the inducement or stimulus to imitate and practice his/her voice.

As smartphones are widely used by people during daily living, the basic necessities of life deal with smartphones or tablet PCs for a certain part. As smartphones provide powerful functions, and the internet connection is faster, the service contents that are provided by smartphones are diversified and easy to use. Therefore, hearing-impaired people often use smartphones as an effective and convenient platform for communication in daily life, learning, or assistance in converting voice to text. Smartphones help create a friendly environment with no barriers.

The purpose of this study was to resolve the problem for a hearing- and speech-impaired student of understanding what a teacher is saying during class in the first place. The investigation was on the “development of a class learning assistive system based on user experience”. The goal was to investigate whether a hearing- and speech-impaired student could understand the class contents from the voice-to-text results that were provided by the learning assistive system. The observation and interview results will also serve as a good reference for design improvement. Additional research subjects were interviewed for their user experience after they used this system. It is expected that this system will find extensive application or will be included into the teaching aids for special students.

The purpose of this study is summarized as follows.

• A case study was carried out on a hearing- and speech-impaired student in her junior year in Taichung, Taiwan, by an investigation of user experience and the use of semi-structured interviews. A class learning assistive system was designed on the basis of the investigation results.

• The content of this learning assistive system includes the real-time voice-to-text conversion of the teaching during class, so that the student can understand the class contents via the resulting texts without the need of others to interpret. It also includes using the device to voice, so that the student can discuss with others in a more convenient way. The device also provides the function of saving the text contents after school, so that the student can study afterward.

• The learning assistive system was reviewed in order to determine whether the learning satisfaction is enhanced for the research subject.

2. Literature Review

The key topic of this study was an analysis of the relevant literature and the investigation of the rationality, essentiality, and innovation of this study. This part includes four sections, as follows. Section 1 provides an analysis of and conclusion on the assistive tools for hearing-impaired and speech-impaired people. Section 2 outlines the collection of data related to voice recognition systems. Section 3 includes the research and investigation of the user experience. Finally, Section 4 contains an exploration of the application of design thinking approaches.

2.1. Assistive Tools for Hearing-Impaired and Speech-Impaired People

Hearing and speech impairment aids are one of the strategic choices to help and improve life and learning of hearing- and speech-impaired people. Designers can use the human activity assistive technology (HAAT) model or the match person and technology (MPT) model to design an aiding tool and follow the WHO’s International Function, Disability, and Health Classification (ICF) system for its evaluation, Finally, patients’ mobility and participation, as well as environmental and personal factors, must be evaluated. Liu [4] conducted a general design study on a multi-function mobile phone for the hearing-impaired. The study pointed out that the use of a mobile phone newsletter is the most common and frequently used communication method for the hearing-impaired [5,6,7]. In Lin [8] study, an Application (APP) system that assists hearing-impaired persons who cannot wear hearing aids in their life needs was designed on the basis of user experience. This is a new auxiliary method that includes voice communication, call to me, signal monitoring, and emergency help.

In the study of Liu [9], an auxiliary communication system was designed for severe hearing-impaired students attending the National Middle School to conduct research on the communication and expression of the student. The results of the study showed that severely hearing-impaired students, with the assistance of the auxiliary communication system, achieved results in sentence structure learning, average sentence length learning, and communication expression. In addition, in the study of Lu [10], the auxiliary communication system was applied to students with severe autism who were attending advanced vocational schools, to conduct research on the effectiveness of classroom participation. Similarly, it was confirmed by the research results that the learning of students with severe autism could also be improved through the auxiliary communication system. On the basis of the above relevant research literature, communication learning is very important for the study and daily life of special students. However, related research focusing on the introduction of autonomous learning aid systems in schools has not been sufficiently discussed. Therefore, by focusing on the hearing-impaired and targeting the language barrier, we propose to design and develop a self-learning system that is suitable for the following learning situations: self-operation of lectures, smooth discussion between students, after-school reviews, and data collation. Students will be of great help for this study. In today’s wireless network era, through the smart phone platform, there is quite a lot of opensource software suitable for people with hearing disabilities, which can help text-to-voice or voice-to-text translations and complete fast and effective communication. Special project design systems are expensive, but if these are integrated into existing software and hardware through ingenuity, with appropriate teaching measures, a simple, highly flexible, and fully functional teaching-aiding autonomous system will be constructed. Its price and availability will be greatly improved. Therefore, the innovative design of this research is based on: first, the wireless network environment, secondly, the use of multiple smartphones and Bluetooth wireless microphones connected to the sound source, and finally, the use of a specially designed APP interface to integrate the automatic sound-to-text or text-to-sound translation of an open-source software and the construction of a set of classroom-independent learning assistance systems. The system is characterized by the combination of common appliances (for example, smart phones, wireless microphones, etc.), open-source software, and self-designed APPs as well as by equipment price parity, high flexibility expansion, and independent design and maintenance.

The technological development of assistive devices in Taiwan presents these features:

• A rigorous language translation/conversion is required for hearing and speech impairment technological assistive devices that are developed in other advanced countries, before they can be introduced for domestic applications.

• The domestic market of assistive devices is small, and this presents a limit for their industrial development.

• The development of assistive devices for special groups lags behind, since there are few design and integration efforts and therefore very few cases, with a high level of heterogeneity. Therefore, the little development of assistive devices for hearing- and speech-impaired people is very limited [11].

As a result, the Taiwan government has been promoting the Welfare Law for People with Disabilities and relevant laws for the protection of disabled people. The government also established the Hearing-and-Speech Assistive Device Promotion Center in order to enhance Taiwanese people’s understanding of assistive devices. This organization also provides consultation to hearing-and-speech-impaired people, with integrated information.

In 2003, the Ministry of Education also entrusted the National Kaohsiung Normal University with the task to set up the Center of Assistive Learning Devices for Hearing-and-Speech Impaired College Students. Its mission is to implement the use of assistive devices by hearing-and-speech impaired college and high school students [12]. The purpose is to help handicapped students use assistive devices adequately during school, so that they can overcome their disability, immerse in school life, and satisfy their right of learning. As specified in the Special Education Law, schools need to provide relevant assistive devices and other services such as space with no barrier for the learning of handicapped students. The goal is to help students complete their learning and participate in school life. During different stages of learning, the organizations that are responsible for the learning of handicapped students are also different. During pre-school and compulsory education, assistive devices and services are provided by special resource centers in each city and county. During the high school collage stages, assistive devices are provided by assistive education device centers that are under the responsibility of the Ministry of Education [12]. Cheng [13] proposed assistive communication systems for the analysis of the overall communication effects and learning effects in hearing-impaired students with autism and disabilities. Tsai [14] used an assistive communication system and script intervention plan to improve the effect of daily dialogs in adults with aphasia. The results indicated that the script intervention plan led to significant improvement in sentence structure accuracy, voice speed, and efficiency. Lin [15] built a database of acoustic models by stratifying syllables, and this system allowed users to communicate with clients by the RESTful application program interface (API) using adjustable modules.

The assistive devices that are provided by the Center of Assistive Learning Devices for Hearing-and-Speech-Impaired College Students to help speech-impaired people in learning and communication function with hearing aids. Our case study is a college student with severe hearing impairment, who cannot wear a hearing aid. She learned to talk when she was young but lost linguistic knowledge. As a result, she needs a system that can convert voice into text for reading and text into voice for the communication with others. Therefore, it is clear that the assistive devices that are provided by the Center of Assistive Learning Devices for Hearing-and-Speech-Impaired College Students are not adequate for this case study.

2.2. Voice Recognition System

The Google App Inventor for Android is for people who have no programming experience and serves to code apps for Android smartphones [16]. It is an open-source software and has been maintained by the Center for Mobile Learning, MIT. It uses visualized objects for stacking up codes. Wolber [17] indicated that App Inventor 2 allows students outside of the information technology sector to devote themselves to programming and learning, so that programs can connect and interact with real life. Therefore, App Inventor 2 was used in this study for the application of an assistive learning system. App Inventors utilize the built-in Google Voice for speech recognition, so that people can avoid typing messages when they are busy. The artificial neural network algorithm for deep learning is used for recognizing user voices [18]. The system can recognize a variety of languages or dialects, such as Mandarin, English, or Korean. Li [18] also proposed the numerical processing of vocabularies by Google Voice and the conversion of the results into vectors by machine learning. The output is also in the form of vectors in order to ensure data accuracy. The internal configuration of the computers can be adjusted in order to ensure the accuracy of translation.

The emphasis for voice recognition is not only on volume, but also on the spectrum of sounds. The relay time is also one of the issues. Therefore, the combination of all these factors make the creation of these devices very complex. Moreover, the variation in the distance from where the voice is coming causes problems for voice recognition [19]. When a voice recognition system receives a sound signal, it applies deep learning knowledge including symbols to recognize which vocabulary is adequate and to comply with the meaning, so that the context can be maintained and the most similar text to the spoken one is generated [20].

Some of the free voice-to-text apps were compared and reviewed. Huang [21] proposed that voice-to-text conversion is more accurate for sentence-by-sentence recognition than for the conversion of a whole speech. When recognizing Taiwanese Mandarin, most devices are still unable to recognize punctuation. Therefore, afterward rectifications are still required. In a literature review, it was found that Python has legibility, since Google has been extending its function library by deep learning. On the other hand, Tensorflow is an open-source algorithm framework that was developed by Google in 2015 to support various types of deep learning calculations [15]. A comparison of three top voice-to-text applications including Google Voice, Apple Voice, and Microsoft Translator indicated that Google Voice has performant functions and a high rate of usage and is free to use.

Besides, when selecting the most appropriate application for this study, it was also found that the APP Inventor 2 allows users to design the desired contents by themselves with the built-in Google Voice. Since Google Voice is the most convenient and easy-to-access resource with a large database for deep learning, it was selected as the voice processing system in this study.

2.3. User Experience Design

According to the definition of the International Organization for Standardization (ISO) in the ISO9241-210 document, user experience (UX) deals with human-oriented designs which allow users to provide demands and the direction of improvement depending on their mood and perceptions when using a product. This approach allows the designer to correct the product and enhance users’ willingness of using this product. This approach also includes features of operation, experience, and emotion during the human–machine interactions.

A user experience represents the users’ emotion and mental perception of a product. User experiences include the attention to a product’s commercials, purchasing behavior, process of using the product, brand image, service, etc. A user might change his/her behavior due to a change in perception.

J.J. Garrett proposed five elements in the user experience of a website in his book “The Elements of User Experience” published in 2002, as shown in Figure 1. People usually think pioneers in the market are advantaged in the development of websites. Therefore, the website context is better understood by dividing it into five elements, as follows: surface, skeleton, structure, scope, and strategy.

If the design of a website complies with users’ perception, users’ willingness of visiting this website will increase, and the number of visits will soon increase too. In other words, these planes indicate the users’ experience of the website. Without these planes, users lack this type of experience. Therefore, it is necessary to provide a user experience design for reference [8]. In fact, user experience can be described as the response to someone asking how one feels when using a certain product.

In 2004, P. Morville extended Information Architecture (IA) to UX and developed a new diagram of user experience, which explained why it is necessary to help users in order to increase the usability of a product. He proposed seven characteristics of the product, which are: useful, usable, desirable, findable, accessible, credible, and valuable.

As shown in Figure 2, the user experience honeycomb can be used to hit the optimal location from multiple targets. First of all, analyzing usability can help people understand the most important features of UX, i.e., which type of website is suitable or easier to acquire information from? How to make users believe the information provided? If the content satisfies users’ expectations, the users will be more confident in their selections. Secondly, a website can be modularized. Finally, each principle in the honeycomb can be viewed as a component that can be modified to change the users’ responses, allowing designers to explore new ideas beyond the conventional ones.

Lin [8] proposed in his study that user experience design can help researchers determine the demands of research targets and the feasibility of designs. Su [22] also proposed that human-centered designs are closely linked to user experience, so that a design satisfying human demands can be created. Lu et al. [23] proposed a user-centered approach for the improvement of existing training tools for surveyors, which is called SimuSurvey. Since few users participated in the preliminary development of SimuSurvey, many teachers and students were suspicious about the innovative application of SimuSurvey to real survey classes. In order to solve this problem, an approach of iteration and increment for a user-centered re-design and development of the tool was proposed in this study. Wang et al. [24] proposed a solution that allows designers to process the demands and messages from dynamic users by demand assessment and predictive methods. They also proposed a knowledge-based concept that is user demand-oriented and based on the structural model of four planes. This approach pays special attention to knowledge collaboration and message interchanges. Hussain et al. [25] and Park et al. [26] proposed user-centered assessment principles for immersive virtual reality (VR) experience or an experience operation interface. Chou [27] proposed a psychological measurement of a user experience model by using the fuzzy measurement approach. The user experience questionnaire (UEQ) was used as a psychological measurement tool to collect evaluation data of subjects. The evaluation variables were converted into Gaussian fuzzy numbers and were aggregated by the aggregation operation. The user experience index (UXI) was developed for a quantitative evaluation of the UX quality. He also carried out a demonstration study of the accidental UX measurement of touching the mouse. The purpose of this study is to develop a class learning assistive system based on user experience which can meet the needs of speech-impaired students. The experiment was carried out considering the user experience so to obtain a system complying with the requirements of being desirable, usable, and useful. The design of the assistive learning system was then improved on the basis of further feedbacks.

2.4. Design Thinking

The concept of design thinking can be traced back to the 1960s when it was called design science [28]. In the 1980s, the design thinking approach extended to the recognition model, which describes the attitude and thinking during the design stage so that a product can have a better design [29]. In 1991, the IDEO design house was founded by the merging of David Kelley Design, London-based Moggridge Associates, San Francisco’s ID Two, and Matrix Product Design with its headquarters in San Francisco. As a well-known design company, IDEO has developed internally and gradually worldwide the design thinking approach [30]. In 2008, IDEO CEO Tim Brown mentioned in Harvard Business Review that “Thinking is just like a designer that changes your product, service, or even strategy”. IDEO designers have different backgrounds, such as in the development of human–machine interfaces, electronics development, software design, and interface design. The most notable design works are Apple computers and the first Microsoft mouse. Design thinking leads designer to new thinking ways and ideas, so that they can find optimal designs by breakthroughs in design conceptions. Brown [31] mentioned that design thinking originated from training designers with professional skills, so that things that are desired by people can be integrated with technology, and more new products can be created.

Design thinking also deals with a design approach that is motivated by human desires. IDEO has been promoting design thinking so that it has become the modern mainstream design approach. Brown [31] also mentioned the use of three design thinking spaces which interact with each other. These three spaces include inspiration, ideation, and implementation, as shown in Figure 3.

• Inspiration deals with looking for a solution to the problem.

• Ideation generates a solution and verifies it.

• Implementation pushes the product forward to the market.

This approach requires cycles of this process until a new direction for a product is reached. The number of cycles between inspiration and ideation might be considerable, until a new product is created.

There are constraints within these three spaces, so that design thinking can be effective. If there is no constraint, the inspiration might become unbridled, and therefore, it is a usual practice to introduce some important constraints to ensure success. The principles of ideation, including feasibility, desirability, and viability, can be applied, as shown in Figure 4 [31,32].

During design thinking, these three principles should be considered continuously, so that a product can reach balanced and standard qualities. However, an in-depth design thinking is required to determine the demands of human beings and understand the real needs of people so that their problems can be solved. This is the motivation that stimulates design thinkers to work on innovative developments rather than on producing designs on the basis of profit figures. However, determining the real needs of people requires the observation of how people actually use a product and of the reasons why they do not use it. This approach requires sympathy, and the actual problems people encounter can only be understood by perceiving and experiencing a product with people who normally use it.

Therefore, Brown [32] proposed that the human being should be the center of design thinking, which, therefore, should lead to a human-centered design. People do not really know what they need, and design thinking can help people find the product they want. There are three elements of design thinking when seeking what people want: insight, observation, and empathy.

Stanford Design School divided the design thinking process into five steps, as shown in Figure 5.

• Empathy: observing users’ behavior, interviewing users, and experiencing like a user.

• Define: describing the observation by sympathy, considering insight and demand.

• Ideate: starting with the development of an idea and identifying the most appropriate solutions for the users.

• Prototype: turning the results of ideation into tangible products or interfaces.

• Test: allowing users to operate on their own and correcting errors in a continuous manner, in order to find the most appropriate product for the users.

Chan [33] proposed that the first step of a design is to have empathy for users’ needs and to identify the problems they encounter and their needs. Wang [30] also proposed the application of the five steps of design thinking to figuring out problems and confirming the issues that need work. Ideation and discussion are also required to reach the final state of a product. Hsiao [34] also proposed that the spirit of design thinking does not lie in design but in thinking. Thinking design is the starting point to enhance creativity. Scholars in various fields such as education and psychology also discussed the correlation between thinking and creative power.

Parnes (1967) proposed the teaching model of creativity problem solution (CPS) [35]. He emphasized the use of systematic thinking to resolve a problem. He also proposed that decision-makers should consider viable approaches from different angles of view. The strategy proposed by Chen [36] is composed of five steps which include understanding facts, discovering problems, paying attention to concepts, finding solutions, and seeking acceptance. Nagel et al. [37] introduced the goal of the James Madison University School of Engineering Approach. In this study, we introduce our course integration design and a sustainable method along with the tutorial method that was used during the entire course. In the design field, Meinel et al. [38], 2011; Jobst et al. [39]; Carayannis [40]; Serrat [41]; Garbuio and Lovallo [42] carried out relevant studies on the application of design thinking. However, there are few studies of the application of design thinking to tutorials or textbooks and their effect on students’ learning. Tu et al. [43] proposed improving tutorials by design thinking. At the empathy stage, interviews and training during class can promote students’ participation. This provides students with substantial help during the interviewing process. It also supplies information of the target group’s demands, strengthens students’ discussion on design-related topics, and builds a favorable educational atmosphere. In addition, it promotes the active interaction between students and teachers, so that students can be more concentrated during class and benefit from the tutorial method.

It is known from the above-mentioned literature review that the current assistive learning devices can be classified as devices for hearing impairment or for speech impairment. Currently, no relevant integrated assistive tool is available. During the preliminary product design stage, it is required to understand users’ demands and needs. Therefore, a semi-structured interview was carried out in this study. The design thinking approach was used to define steps for the design planning. During the entire design process, view from the users’ standpoint and use of the user experience were required to verify whether the assistive learning system could meet users’ demand.

3. Research Methodology and Process

The first step of this study was to observe the research subject during class. The purpose was to identify her problems and then carry out interviews with her. After the demands and problems of the subject were defined, the design thinking approach was used to determine the most appropriate assistive learning system for the research subject. A learning satisfaction scale was used for follow-up improvement based on the subject’s feedback, so that the problems that appeared during class could be determined. The final step was to obtain an overall feedback from the user experience questionnaire. In this way, the problems that the research subject encountered during class can serve as the basis for the development of communication assistive devices. Firstly, the preliminary natural scenario observation method was used to observe the research subject’s condition during class and identify her problems. After the class, a semi-structured interviewing approach was implemented in order to understand the subject’s needs. After the identification of all relevant problems, a learning design was created. During the middle stage of the design, the assistive learning system was integrated into the class, so that the research subject could try it. In addition to observing the system usage by the research subject during class, an interview was carried out after each experimental period. The purpose was to obtain constant feedback on the usage for the correction of the assistive learning system. The final stage was to introduce the modified assistive learning system into the class in order to observe and then understand through an interview whether the research subject was helped by the system during class. In this study, design thinking and user experience can be divided into three stages, which include the user observation stage, the user experience feedback interview, and the modification of the assistive learning system followed by a feedback test. These stages are described, respectively, as follows.

The first stage dealt with the study of the user experience. In this stage, the natural scenario observation method was used in combination with semi-structured interviews. The purpose was to find the research subject’s problems during class by observation. Three interviews were carried out to help the researchers understand the problems that were encountered by the research subject during class. After collection, the problems were analyzed, and the learning experience of the research subject together with her classmates was observed. The solution to the communication problem was also determined along with the problems that were encountered during class. After the interviews, it was determined to develop an assistive voice-to-text tool that could help the research subject to receive the class contents.

The second stage consisted in adopting the design thinking rules for designing the assistive learning system. After the observation and interviews of the first stage, the subject was provided with the assistive learning system which could perform voice-to-text processing, so that the research subject could receive class contents from a microphone. She could obtain knowledge in real time during class, without the need of translation by her classmates. During class discussions, she could also ask questions using the text-to-voice function. Others could use the microphone to answer directly. During class, a smartphone with a recorder function or a voice recorder could be placed near the teacher in order to take recording. This approach allows the research subject to review lessons after school and to learn in an autonomous way.

The third stage dealt with the user experience test. The assistive learning system that was designed in the second stage was implemented, so that the research subject could participate in a specific research test. The problems that arose during the test and the questions that were brought up during the interview were recorded. The research subject was also asked to fill in a learning satisfaction questionnaire. The learning satisfaction questionnaire and related scale that were developed in this study allowed us to understand the subject’s learning satisfaction when using the assistive device. The problems that arose during the test and the questions that were brought up during the interview were used to improve the assistive learning system. After that, the research subject was asked to carry out the test again in order to verify whether her learning satisfaction was improved with respect to the previous round of experimentation.

Design thinking and user experience were implemented in combination in this study. The research subject’s demands regarding the assistive device design were investigated by observing and interviewing the subject on her user experience. The design thinking approach was implemented in the design of the assistive learning system. During the design process, it is required to include the user experience elements of desirability, usability, and usefulness. The problems that were encountered during the field test were recorded to further improve the system. The goal was to determine whether the design of the assistive learning system met the research subject’s demands, so that the assistive learning system could really be useful to the research subject during class. The flowchart of this study is shown in Figure 6.

3.1. Case Study

The research subject in the case study is a hearing-and-speech-impaired college student in her junior year in Taichung, Taiwan. After interviewing the subject, it was known that the subject suffered from a bacterial infection in the throat cartilage in senior high school and was able to speak after a tracheostomy operation. After the operation, she gradually lost her hearing. During the sophomore year in college, she lost her entire hearing capability. The doctor determined that she lost hearing due to an ear stroke, which eventually turned into a heavy hearing impairment. The subject was determined to treat her hearing impairment. After hearing loss, she tried assistive devices in the attempt to regain hearing. However, the doctor determined that her hearing loss had turned from minor to severe, so that she was not able to wear any hearing aid. This caused challenges in her learning compared to other students’. Moreover, after she lost hearing, she was unable to practice pronunciation and therefore she gradually forgot how to speak. She gradually learned how to read lip language in order to understand people talks. However, since she did not follow a classical lip language course, she could presently read lip language only at a slow pace. Currently, the subject keeps returning to the doctor for language therapy. She thus learned how to voice in an indoor environment. Although her voice is still rough, this was a great progress for her. The disease occurred in her senior high school, when she had a fundamental knowledge of the typical language. Therefore, her learning went smoothly than for other inborn hearing-and-speech-impaired people, as she already had a background. During college, she took a year off to recover from a relapse. Although she had such impairment, she did not lose her enthusiasm for learning. After she recovered, she went to school again to study. During class, she still needed others’ help to understand class contents, as she was not able to speak and to keep up with the teachers’ pace by reading their lips. As a result, the school assigned a classmate to accompany her in class, so that the research subject did not need to worry about the learning progress. The teachers also used presentation slides in class or used monitors to switch between systems, so that the subject could see the contents in a convenient way. These methods allowed the subject to make notes and understand better the class contents. Background information on the research subject is shown in Table 1.

The classmate provided the research subject with assistances as follows. If the teacher did not prepare any presentation slides, the classmate interpreted the contents by communication apps, so that the subject could understand the class contents. However, the classmate still needed to learn as well, so she/he might not be able to interpret all the time. The research subject sometimes needed to ask the classmate again in order to obtain the required information. Alternatively, she needed to ask the classmate for supplementary data after class, so that she could make notes on her own in order to complete her learning. From the teachers’ point of view, since the school had assigned a classmate to the research subject, they did not pay special attention to the research subject and usually allowed the classmate to interpret or assist. The research subject also studied the presentation contents by herself with the classmate’s assistance in providing supplements. However, the research subject might lag behind, as her typing might be slow, or the session time might be short. The observation indicated that there was still a gap between teachers’ talk and research subject’s understanding, even with the classmate’s interpretation. The research subject still wanted an assistive tool that could help her learn the teachers’ class contents in real time without others’ interpretation. The interpretation might also present a problem, as there might be a lag in the classmate’s understanding. After further communication with the teachers, an assistive tool that was suitable for the research subject was developed following her observation in class. This tool allows the classmate to attend to her/his own learning, while the research subject can use the assistive learning system to communicate with others and learn on her own after school, using the recordings of the class.

3.2. Data Analysis

Scholars such as Chang (1994) interpreted satisfaction as the intrinsic state and mental perception that are generated during the course of pursuing a goal. Moreover, learning indicates the course of behavioral changes rather than the behavioral performance after learning. The scale that was used in this study referred to the “Study of correlation between the participation motivation and learning satisfaction for the elementary students in New Taipei City” (Yang, 2014) and “A study of the learning satisfaction and learning effects of the gym class for senior high schools—A case study on Taichung First Senior High School and Taichung Girls’ Senior High School” for facilities and equipment design. In the latter study, the participants were asked to assess their satisfaction of facilities and equipment during class. A scale of learning satisfaction regarding the assistive learning system was prepared for this study. When using this questionnaire, after the end of each experiment, the participants are asked to rate their learning satisfaction using the scale. The questionnaire includes two portions, as follows. 1. Background information of the participants. 2. Questionnaire about learning satisfaction when using the assistive learning system, with a total of 13 questions. The Likert five-point scale is used for the evaluation, and the scores are: 5—very satisfied; 4—satisfied; 3—neither satisfied nor dissatisfied; 2—dissatisfied; 1—very dissatisfied. The scale used for determining the learning satisfaction when using the assistive learning system is shown in Table 2. The participants are asked to fill in this table regarding their satisfaction of the assistive device.

3.3. Operation of the Assistive Learning System

The assistive learning system in this study was built by APP Inventor 2 and can be installed on Android systems. The voice-to-text function was developed using this type of open-source freeware, so that the research subject does not need other people to translate the class contents. The teacher was equipped with a Bluetooth wireless microphone for recording. In order to enhance the recording quality of the microphone, the research subject can plug the receiver on the AUX out port of her smartphone in order to get them paired. The built-in voice system is the Google Voice. When pressing down the microphone icon, the Google Voice starts to record and converts voice into text. This way, the research subject can understand the class contents, and the text file is saved in the storage folder after pressing the “Save” button.

The assistive learning system of this study was created by the design thinking approach. For the “empathy” consideration, the research subject was asked to let the researchers observe her for a period of three weeks. The problems that were encountered during class could be determined by observation at class. The research subject was further interviewed, and the difficulties in her learning were determined as follows.

• Without classmates’ assistance or any presentation file, the subject could only learn on her own.

• The subject needed the classmates’ translation, or she could obtain class information through messages from the smartphone.

• Even with presentation files during class, the subject still needed classmates’ assistance with supplemental files after school.

• The group homework was assigned to the classmates, and the discussion was mainly through messages.

By classifying the research subject’s demands, it was found that she preferred to directly receive the teachers’ messages during class without the need of others to translate. During the “inspiration” period, the idea of the assistive learning system was proposed by referring to relevant literature and products. The “prototype” design was created by APP Inventor 2, and the “verification” was carried out at the final stage. New problems were discovered during the verification stage, and the design was rectified. The flowchart of the design thinking process is shown in Figure 7.

The process logic of the assistive learning system of this study is shown in Figure 8. The purpose was to find the problems during class and after the discussion with classmates, for further rectification.

The interface design is presented in Figure 9 and Figure 10, which show Google’s voice-to-text button. A text hint was also created to inform the research subject of pressing the microphone icon before using this recording function. The teacher wore a Bluetooth wireless microphone for one-to-one connection. The receiver was plugged into the AUX in port of the smartphone for receiving the voice. This was for the convenience of the research subject, allowing her to understand class contents via the voice recognition function without the need of others’ translation.

4. Discussion

The purpose of this study is to investigate whether the assistive learning system for the hearing-and-speech impaired people can enhance the learning satisfaction for the hearing-and-speech impaired students. In this section, the experimental results were analyzed and investigated in order to summarize the research problems and the design concerns that should be taken care of. This section includes four sub-sections as follows. 1. The implementation of the assistive learning system. 2. The design course of the assistive learning system. This deals with the use of user experience to develop a system that can be used by the research subject during class. 3. Introducing the assistive learning system into the course of a class and collecting the problems that were discovered in the experimental field during the process of the experiment. The results were used for further rectification in order to meet the research subject’s demand. 4. The hearing-and-speech impaired student’s change due to the introduction of the assistive learning system was recorded. It is expected that the introduction of this new system can bring up a different class style so that the research subject can open her mind and overcome the mental barrier in her learning.

4.1. Implementation of The Assistive Learning System

There are three topics in this section, i.e., tutoring in the classroom, discussion with classmates, and review after school. The observation record, interview record, video-recording, and feedback data were collected for the understanding of the research subject’s condition of learning. The results served as a reference for follow-up system improvement.

4.1.1. Tutoring in the Classroom

In this study, there were three times of observation of problems and three times of observation of the usage conditions which include two times of problem rectification and the final observation. The problems that were encountered during the three times of observation of the usage condition and the improvement after the real test are listed below. The researchers took videos and photos of each of the observations along with side notes as records.

After further discussion with the research subject, the researchers asked for a teacher’s permission to carry out the experiment during class. For the three times of problem observation as shown in Figure 11, the researchers sat at the neighborhood to the hearing-and-speech impaired student for observation. The classmate sat by the side of the hearing-and-speech impaired student and translated the class content for the hearing-and-speech impaired student. By observing the condition of the hearing-and-speech impaired student at class, the observed problems were recorded as the basis of the development of the assistive learning system. The researchers took photos and videos for record.

During the first stage of the experiment, the observation indicated that, due to the speech speed of the teacher, the research subject might miss what the teacher was talking about when she still looked at the previous section of translated texts. As a result, she might not be able to connect the current content to the previous one. After the experiment, the researchers reviewed the problem by interviewing the research subject and found that the voice-to-text results should be displayed two sentences at a time so that the results appeared alternatively. This serves as the basis of the second round of improvement. In addition, since the teacher’s speech speed is normal, the voice-to-text results presented no problem. However, sometimes when they were viewing videos, the research subject might be confused as there was not subtitle.

In the second round of improvement, the observation from the first round of experiment proposed the use of two voice-to-text screens at a time. As compared to only one voice-to-text screen in the first experiment, it made the research subject easier to read. Originally, a wired microphone was used and the teacher needed to take the microphone so that he sometimes forgot to take the microphone. Therefore, the microphone was fixed on the stage in front of the teacher. However, since the distance varied and the microphone wire was too long, the translation speed and stability was not good enough for a precise translation. After the first experiment, the research subject proposed the use of two voice-to-text screens instead of only one. Moreover, the screen size was also increased. Therefore, the operation should become easier. The wired microphone was also replaced by a wireless one in order to enhance the recording stability and the translation stability. However, the speech speed in the second experiment was faster than the first one so that the translation of the second experiment was still not as good as the first one.

The observation in the third experiment indicated that, the research subject was much more familiar with the operation interface. Therefore, the operation in the second experiment was faster than in the first one. In the third experiment, the wireless microphone was worn by the teacher so that the receiving stability of the voice-to-text results was greatly enhanced than the previous experiment with wired microphone. Since the speech speed of this teacher was faster than the previous ones, the translation result was not as good as the first one. However, the microphone was closer to the teacher’s mouth and therefore the voice-to-text result was better than the second experiment.

It apperead from these three experiments that many factors affected the experiments, so that many problems arose. For example, there was an air conditioner in the classroom for experiment and the noise was from above the teacher’s location. As a result, the microphone was interfered during recording. Moreover, the talking volume from the classmates, the teacher’s gesture of using the microphone, the speech speed, and the internet connection stability are some of the factors that affect the receiving stability. These factors also affected the translation stability. The researchers improved the conditions by replacing a wired microphone with a wireless one and a Bluetooth receiver, controlling the environment for experiment, carrying out the experiment in a quiet classroom, choosing a location that was far away from the noise source, and changing to a location with better internet connection. It was found that the recording stability was better under controlled environment than an uncontrolled one and the accuracy was enhanced as well.

4.1.2. Discussion with Classmates

The purpose of the discussion with classmates is for the convenience of further discussions. Therefore, the voice-to-text translation is required for the class so that the classmates can used to voice function for communication. The research subject requires the conversion text into voice so that the discussion can be easier by allowing the classmates to listen to the discussion contents without the need to look at the texts on the smartphone for discussion, which caused some lags during transmission. By interviewing the research subject for the perception of the system, it was found that this approach was more convenient when carrying out discussions. Although typing consumed some time, there was no need to look at the smartphone to understand the conversation. Moreover, they can look at each other during discussion as a typical way of discussion between classmates as shown in Figure 12. For a discussion between multiple students, two smartphones were prepared with one for the connection to a projector and the other one for the use by the research subject. When responding to a question, the voicing function was used.

4.1.3. Review after School

For an average student, it is easy to take notes during class and use his/her ears and eyes to hear and see the class contents or other external information. However, the hearing-and-speech-impaired student in this study required assistive devices to absorb the class information and also needed to take notes but was unable to keep up with the other students’ pace. Therefore, after discussion, the researchers determined to save the translated texts to a storage folder. The research subject needed only to press the “Read” button to access to the class contents. This approach allowed the research subject to carry out reviews after school and take notes as records.

4.2. Intervention of the Assistive Learning System into the Courses

The research subject used the assistive learning system to carry out three class experiments and one experiment under a controlled environment. She was then asked by the researchers to give her feedback in the learning satisfaction questionnaire. It was found from the results of learning satisfaction in Table 3 that a difference existed between learning in the average class and in a class in a controlled environment. The statistics of the line chart in Figure 13 indicates the difference between these four experiments.

The scores of Questions 2, 7, and 11 in the learning satisfaction feedback questionnaire for the first experiment were lower than the others, and the reasons for this are summarized in Table 4.

During the first time of using the assistive learning system, the research subject needed to practice several times and she felt a bit discouraged during the initial intervention. The teacher sometimes placed the microphone on the table, and the recording quality was not good. Sometimes the classmate still required to translate for the research subject’s understanding, so that she could be able how to study after class. The second experiment was carried out after these problems from the first experiment were rectified. The feedbacks indicated that the scores of Questions 2, 4, 7, 8, 9, and 11 were lower than those of the others. The reasons were clarified after further interviews, as shown in Table 5.

During the implementation experiment, the teacher was allowed less time to hold the microphone, and there was not clear description of the presentation contents, therefore the research subject was a bit confused.

In the third experiment, the research subject was more familiar with the overall operation, and the microphone was attached to the teacher clothes for a better recording quality. However, the internet connection in the classroom was not stable, the volume of the teacher’s speech was low, and the recording quality was affected by the teacher’s pronunciation. As a result, translation by tahe classmate was still required. A wireless microphone was used in the third experiment, and the leaning satisfaction feedback is shown in Table 6.

It was found from the above-mentioned experiments that external factors affected the research subject’s learning satisfaction during usage. Therefore, a further experiment was carried out in a classroom with a controlled environment in order to observe whether the research subject’s learning satisfaction was enhanced. During the experiment under a controlled environment, the research subject sat away from the air conditioner, and the Wi-Fi connection was enhanced within the classroom. It was found that the feedback values under the controlled environment were positive as compared to those in the previous three experiments. The results indicated that the recording quality could be enhanced when the background noise in the classroom was reduced. The stable Wi-Fi connection also contributed to better voice-to-text results, and therefore, the location of the classroom is also important. The classroom for the previous three experiments is located at the corner of the Tutorial Building. On the contrary, the classroom with the controlled environment is located at the center of the building with a stable Wi-Fi connection. Several factors affected the research subject’s feedback on the system’s usage. Question 13 of the feedback questionnaire of the assistive learning system had a higher score under the controlled environment than in the other three experiments. After a further interview, it was found that under the condition of controlled external factors, the stability of recording and the internet connection both affected the research subject’s scoring of the assistive learning system. The feedback information served as a reference for improving the assistive learning system, so that the research subject can find the system practical and convenient.

After the end of all experiments, the research subject was asked to answer a questionnaire survey about her general user experience of the assistive learning system. As shown in Table 7, the questionnaire referred to the “Application of user experience to the research and design of apps for the deaf-mute life” by Lin [6]. The purpose was to determine whether the design of the assistive learning system complies with the seven principles that were proposed by P. Morville for user experience.

The results of the experiment are shown in Table 8. The results indicated that a simple presentation of the interface and clear hints on how to operate the system allowed the research subject to use the assistive learning system on her own. The research subject was more interested in continuous learning, since she needed less help from others for her learning. Although she still needed other assistive tools during class, she progressed faster on learning as compared to the earlier way of asking for classmate’s help for learning. The reasons of lower scores for Questions 4–8 are shown in Table 9. The overall usage perception was still affected by the external factors.

4.3. Changes in the Research Subject’s Learning Experience Due to the Intervention of the Assistive Learning System During Class

The research subject was asked to fill in a questionnaire and was also interviewed after the end of each experiment. It was known from each of the interviews that the assistive learning system was helpful for the hearing-and-speech-impaired student during class. Therefore, after the interviews, it was found that the improvement of the research subject’s learning experience after the assistive learning system intervened into the class was fourfold, as follows.

• The research subject could attend the class and study by herself by reading the texts.

• The research subject could get firsthand class contents without any delay due to the interpretation by others.

• There was no need for the research subject to rely on other classmates or feel any pressure during learning.

• As compared to the original way of studying without the assistive learning system, the research subject was able to study and take notes independently.

The assistive learning system offers not only an external support but also encouragement to learning from the psychological standpoint of the research subject. It was known from the experimental results and the interviews to the research subject that she could learn by herself and was happy to learn. Originally, she needed other classmates’ help to finish her schoolwork. With the assistive learning system, she could learn in her own way. As a result, her mind gradually opened, and she was willing to interact with more classmates. This might seem common for the average students. However, it is not easy for a hearing-and-speech-impaired student as the research subject is.

The research results also indicated that the learning satisfaction of the research subject was enhanced after the use of the assistive learning system. During the experiment, in order to meet the research subject’s needs, the approaches of design thinking and user experience were implemented, and the results of each experiment were evaluated using rating scales. The interview records were also analyzed, and conclusions were made in order to understand the research subject’s problems after usage of the system. The assistive learning system was improved and re-designed accordingly. Therefore, after the intervention of the assistive learning system that was developed in this study, the research subject became more optimistic and positive, relied less on others, and couls study independently.

5. Conclusions

The research results showed that the learning satisfaction of the research subject after the use of the assistive learning system was enhanced. During the experiment, in order to meet the subject’s needs, the approaches of design thinking and user experience were implemented. A questionnaire survey was carried out, so that the experimental results were evaluated by rating scales. The interview records were analyzed and summarized in order to identify the problems that occurred during usage. The problems were immediately rectified, and the system was re-designed. As a result, the research subject became more optimistic and positive after using the assistive learning system. She relied less on others and started to learn independently.

The investigation of this study was accomplished on the basis of semi-structured interviews, design thinking, and user experience. The experiments were carried out by following the five steps of design thinking. An assistive learning system that complied with the research subject’s needs was created. This system was further improved by the user experience approach in order to meet the research subject’s demands.

The achievements of this study are as follows.

• An assistive learning system that covers three orientations including tutoring in class, discussion with classmates, and review afterschool was created on the basis of the user experience and design thinking approach. This system meets the hearing-and-speech-impaired student’s demands.

• The assistive learning system is composed of open-source freeware, smartphone, Bluetooth microphone, and customized apps. The voice-to-text function that is provided by Google Voice was implemented in the core function of voice-to-text conversion. The practical needs of the research subject were investigated during the design planning stage. The resulting assistive learning system is appropriate, convenient, and easy to use. This system can enhance the subject’s learning in an effective way. A reading function was provided for review after school. This allows the research subject to access the data for study after class and to take supplementary notes that might be missed during class. Moreover, the overall design can be further improved by interviewing the research subject after the experiments. The system can be rectified so that it can get better. The results indicated that this assistive learning system is very helpful for improving the research subject’s learning during class.

• The results of the learning satisfaction scale indicated that the research subject’s learning satisfaction was enhanced significantly after the intervention of the assistive learning system by the gradual improvement of the equipment. Moreover, the subject can also study independently by using this assistive learning system. She only needed the classmate’s help when she encountered terms or figures that could not be converted to text. As she spent more time studying on her own, she gained more confidence in her learning.

After repeated improvements on the system, the final assistive learning system turned out to be adequate for use by hearing-and-speech-impaired students. During the experimental process, user experience was implemented for verification, so that the final system can comply with the demands of hearing-and-speech-impaired students. Some uncontrollable factors such as internet connection’s stability, air conditioner’s noise, and the noise from other classmates were also discovered and could not be expected before the experiment. However, the system was further improved by including these factors into the design consideration. When solving these real problems, specific improvements can be carried out by experimenting in a controlled environment. The results also indicated that the proposed system is helpful for hearing-and-speech-impaired students no matter if in an uncontrolled environment or a controlled one. Therefore, the conclusions of this study are as follows.

• The relevant functions of this assistive learning system can become completer and precise by introducing more professional designs into the system. This system can also be promoted among more hearing-and-speech-impaired students who have similar disabilities to those of the research subject in this study. This system can assist teachers in special schools and can be also extended to other fields outside classrooms such as for the communication of hearing-and-speech-impaired people in daily life.

• It is recommended to include digital technology and professional devices into the system in follow-up studies, so that the assistive learning system can provide hearing-and-speech-impaired students with precise and adequate usage. External factors such as internet connection’s stability, air conditioner’s noise, or noise from other classmates can be overcome. Moreover, the problem of inaccuracy in the voice-to-text conversion or for special terminology of professional subjects is expected to be resolved in follow-up experiments by the introduction of more smart technologies.

• From the aspect of group communication and discussions, more advanced technologies and services that are user-oriented are expected to be introduced. This approach can allow more convenient, fast, and precise communication and discussions, so that this system can be implemented in other professional learning scenarios or daily communication situations. It is also expected that this study can inspire more studies in the field, so that more researchers can contribute to design and create services for social vulnerable groups.

Author Contributions

Conceptualization, J.-S.L. and W.-C.W.; methodology, J.-S.L. and S.-C.H.; software, W.-C.W.; validation, J.-S.L. and W.-C.W. and C.-H.L.; formal analysis, J.-S.L. and W.-C.W.; investigation, J.-S.L. and W.-C.W.; resources, J.-S.L.; data curation, J.-S.L.; writing—original draft preparation, J.-S.L. and W.-C.W.; writing—review and editing, C.-H.L. and S.-C.H.; visualization, J.-S.L. and W.-C.W.; supervision, J.-S.L.; project administration, J.-S.L.; funding acquisition, J.-S.L. and C.-H.L.

Funding

This work was supported by the Ministry of Science and Technology of the Republic of China under grant MOST 108-2628-E-029-001-MY3.

Conflicts of Interest

The author declares no conflicts of interest.

Figures and Tables

Figure 1. The elements of user experience.

Figure 2. User experience honeycomb.

Figure 3. Three spaces of design thinking.

Figure 4. Three principles of design thinking.

Figure 5. Five steps of design thinking.

Figure 6. Flow chart of the research process.

Figure 7. Flow chart of the design thinking process.

Figure 8. Process logic of the assistive learning system.

Figure 9. Homepage of the interface of the assistive learning system.

Figure 10. Voice-to-text recognition.

Figure 11. Observing problems during class.

Figure 12. Discussion between classmates.

Figure 13. Line chart of the learning satisfaction.

Table 1

Background information on the research subject.

Subject	Gender	Age	Impairment Type	Impairment Level	ICD Diagnosis	Impairment Since	Note
T	Female	24	Hearing function impairment (Type 2)	Severe	H90.3 (Voice-sensing nerve hearing loss, dual tests)	Senior high school, diagnosed as severe hearing impairment and unable to wear any hearing aid	Bacterium infection on throat cartilage in senior high school; she was able to speak after a tracheostomy operation. Undergoing language therapy

Source: Abstract from the subject’s anamnesis by the researchers.

Table 2

Scale of the learning satisfaction of assistive learning system.

I think	Very Satisfied	Satisfied	Neither Satisfied nor Dissatisfied	Dissatisfied	Very Dissatisfied
The assistive learning system is very helpful for my understanding of the class contents.
It is easy to operate the assistive learning system during class.
Using the assistive learning system is not helpful for my learning.
Via the assistive learning system, I need no other person to help me understand the class content.
The use of the assistive learning system doesn’t attract me.
The assistive learning system makes it more difficult for my learning.
Via the assistive learning system, it is more convenient for me to receive the teachers’ messages.
Using the assistive learning system can enhance my sense of accomplishment in learning.
The devices of the assistive learning system are convenient for me.
The voice-to-text function of the assistive learning system is helpful for me.
Even if there is no presentation during the class, I can still understand the class contents from the teachers.
Using the assistive learning system can enhance my confidence during class.
I favor the use of the assistive learning system during class.

Table 3

Results of the learning satisfaction experiments.

Number of Experiments	1	2	3	4	5	6	7	8	9	10	11	12	13
First experiment	3	2	4	3	4	4	2	3	3	4	2	3	3
Second experiment	3	2	3	2	3	3	2	2	2	3	2	3	3
Third experiment	4	3	3	2	3	3	3	4	2	3	3	4	3
Experiment under a controlled environment	4	4	4	4	3	3	4	4	4	5	4	4	4

Table 4

Interview results of the learning satisfaction feedback questionnaire for the first experiment.

Learning Satisfaction Question	Interview Result
2. The operation of the assistive learning system is easy during class.	I didn’t know how to use the system the first time when I used it.
7. It is convenient for me to receive the teachers’ messages via the assistive learning system during class.	The internet connection is not stable, and therefore the translation result might show wrong punctuation.
11. I can understand the class contents by the assistive learning system even if there is no presentation during class.	I still need to see presentation slides, so that I can make notes during class.

Table 5

Interview results of the learning satisfaction feedback questionnaire for the second experiment.

Learning Satisfaction Question	Interview Result
2. The operation of the assistive learning system is easy during class.	I didn’t know how to use the system first time when I used it.
4. With the assistive learning system, I don’t need others to understand the class contents.	During the implementation class, the teacher putdown the microphone, and I needed others’ help to understand the class contents.
7. It is convenient for me to receive the teachers’ messages via the assistive learning system during class.	The internet connection is not stable, and therefore the translation result might show wrong punctuation.
8. My learning accomplishments were enhanced by using this assistive learning system.	During the implementation class, the translation was incomplete.
9. The device of the assistive learning system is convenient for me to use.	The wired microphone limited the distance and made the usage inconvenient.
11. I can understand the class contents by the assistive learning system even if there is no presentation during class.	I’m afraid of missing some class information if a presentation was lacking and only the voice-to-text conversion was available.

Table 6

Interview results of the learning satisfaction feedback questionnaire for the third experiment.

Learning Satisfaction Question	Interview Result
4. With the assistive learning system, I don’t need others to understand the class contents.	Sometimes, the recording quality was affected by the teacher’s voice volume, and I still needed the classmate’s help.
9. The devices of the assistive learning system is convenient for me to use.	I needed to tell the teacher that I have a special condition and need assistance in using the assistive learning system.
11. I can understand the class contents by the assistive learning system even if there is no presentation during class.	I’m afraid of missing some class information if a presentation was lacking, and only the voice-to-text conversion is available.

Table 7

Questionnaire of the general user experience of the assistive learning system.

User Experience Question	Very Agree	Agree	Neither Agree nor Disagree	Disagree	Very Disagree
1. The screen of the assistive learning system is easy to operate for me.
2. The functions of the assistive learning system are easy to use for me.
3. I feel the assistive learning system can help me achieve the goal of communicating with the teachers.
4. I will use the assistive learning system to obtain class information from the teacher during class.
5. The assistive learning system helps me to understand the class contents that were delivered by the teacher during class.
6. I will use the assistive learning system to interact with the teacher during class.
7. The voice-to-text function is helpful for my progress during class.
8. I feel that I need no others to help translate the class contents by using the assistive learning system.
9. The assistive learning system makes me more interested in learning.
10. The portability or the assistive learning system is convenient for me.

Table 8

Feedback from the questionnaire of the general user experience of the assistive learning system.

Difference in Usage Experience	1	2	3	4	5	6	7	8	9	10
Score	5	4	4	3	3	3	3	3	4	4

Table 9

Feedback from the questionnaire of the general user experience of the assistive learning system.

Question	Interview Result
4. I will use the assistive learning system to obtain class information from the teacher during class.	The information that was received might be intermittent due to unstable internet connection.
5. The assistive learning system helps me to understand the class contents that were delivered by the teacher during class.	The teacher’s speech volume affected the recording quality.
6. I will use the assistive learning system to interact with the teacher during class.	There was no interaction with the teacher during class.
7. The voice-to-text function is helpful for my progress during class.	It was required to ask the classmate when the voice reception was bad.
8. I feel that I need no others to help translate the class contents by using the assistive learning system.	I still depended on the classmate’s help on the translation of certain terms.

References

1. Li, C.Y. About the Language Education for Hearing-Impaired Students; Special Education Books Taichung Normal School, Special Education Center: Taichung, Taiwan, 2010; Volume 9901, pp. 9-14.

2. Huang, Y.Y. A Study of the Application of Speech Analysis to the Design of Tone Training for Hearing-Impaired People; Southern Taiwan University of Science and Technology: Tainan City, Taiwan, 2017.

3. Changchun Deaf. Deaf is not Equal to Dumb. Are You Aware of the Hearing and Speech Things; 2017; Available online: https://kknews.cc/zh-tw/health/vzaqmxa.html (accessed on 4 September 2019).

4. Liu, C.H. The Universal Design of Multi-function Mobile Phone for Hearing-Impaired Users; National Chiao Tung University: Hsinchu City, Taiwan, 2009.

5. Bakken, F. SMS Use among deaf teens and young adults in Norway. The Inside Text: Social, Cultural and Design Perspectives on SMS; Springer: Dordrecht, The Netherlands, 2005; pp. 161-174.

6. Henderson-Summet, V.; Grinter, R.E.; Carroll, J.; Starner, T. Electronic communication: Themes from a case study of the deaf community. Proceedings of the 13th IFIP Conference on Human-Computer Interaction; Rio de Janeiro, Brazil, 10–14 September 2007; Volume 4662, pp. 347-360.

7. Pilling, D.; Barrett, P. Text communication preferences of deaf people in the United Kingdom. J. Deaf Stud. Deaf Educ.; 2007; 13, pp. 92-103. [DOI: https://dx.doi.org/10.1093/deafed/enm034] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/17602163]

8. Lin, K.T. User Experience Design and Research for the Daily Life of Hearing Impaired Persons; National Taiwan University of Science and Technology: Taipei City, Taiwan, 2013.

9. Liu, H.Y. The Study of Effectiveness of Sentence Expression through Augmentative and Alternative Communication for a Junior High School Student with Severe Hearing Impairment; National Dong Hwa University: Hualien County, Taiwan, 2013.

10. Lu, Y.P. Effects of Augmentative and Alternative Communication Intervention on Classroom Participation for High school Students with Severe Autism; National Chiayi University: Chiayi City, Taiwan, 2014.

11. Hearing-and-speech Assistive Device and Resource Promotion Center. Manual of Hearing-and-Speech Assistive Device and Resource; Ministry of the Interior: Taipei City, Taiwan, 2004.

12. National Special Education Information Network. Center of Assistive Devices for Hearing-and-Speech Impaired Students. 2017; Available online: https://special.moe.gov.tw/article.php?paid=134 (accessed on 4 September 2019).

13. Cheng, Y.W. A Meta-Analysis on the Effects of Augmentative and Alternative Communication for Students with Autism; Taipei City University: Taipei City, Taiwan, 2016.

14. Tsai, C.T. The Study of Applying Augmentative and Alternative Communication and Script Intervention Plan in Improving the Effectiveness of Conversation for Adults with Aphasia; National Taipei University of Nursing and Health Science: Taipei City, Taiwan, 2016.

15. Lin, B.Y. Speech Recognition for People with Dysphasia in Chinese; National Taipei University: New Taipei City, Taiwan, 2018.

16. App Inventor Overview. Developing Android Apps with MIT App Inventor 2; GrandTech Information Co. Ltd.: Taipei City, Taiwan, 2014; pp. 4-5.

17. Wolber, D. App Inventor and Real-World Motivation. Proceedings of the 42nd ACM Technical Symposium on Computer Science Education; Dallas, TX, USA, 9–12 March 2011; pp. 601-606.

18. Li, C.F. Google Teaches You Deep Learning; Taiwan Mansion Books Group: New Taipei City, Taiwan, 2017.

19. Liao, H.C.; Kuo, C.C.; Lin, C.H. Challenges and Preliminary Study on Indoor Distant Speech Recognition. Comput. Commun.; 2015; 164, pp. 42-54.

20. Chen, Y.W. A Study of the Language Model Adaptation by Using Vocabulary Characteristics on the Voice Recognition during Meeting; National Taiwan Normal University: Taipei City, Taiwan, 2018.

21. Huang, E. Three Free Online Voice-to-Text Tools for Real-Time Notes in Chinese. 2016; Available online: https://www.playpcesor.com/2016/07/speech-to-note-txt.html (accessed on 4 September 2019).

22. Su, W.T. Exploring the Motivation of the Players in Real Escape Room with User Experience; National Yunlin University of Science and Technology: Yunlin County, Taiwan, 2019.

23. Lu, C.C.; Kang, S.C.; Hsieh, S.H.; Shiu, R.S. Improvement of a computer-based surveyor-training tool using a user-centered approach. Adv. Eng. Inform.; 2009; 23, pp. 81-92. [DOI: https://dx.doi.org/10.1016/j.aei.2008.07.001]

24. Wang, Y.H.; Yu, S.H.; Xu, T. A user requirement driven framework for collaborative design knowledge management. Adv. Eng. Inform.; 2017; 33, pp. 16-28. [DOI: https://dx.doi.org/10.1016/j.aei.2017.04.002]

25. Hussain, J.; Hassan, A.U.; Muhammad Bilal, H.S.; Ali, R.; Afzal, M.; Hussain, S.; Bang, J.H.; Banos, O.; Lee, S.Y. Model-based adaptive user interface based on context and user experience evaluation. J. Multimodal User Interfaces; 2018; 12, pp. 1-16. [DOI: https://dx.doi.org/10.1007/s12193-018-0258-2]

26. Park, W.J.; Heo, H.Y.; Park, S.J.; Kim, J.M. A Study on the Presence of Immersive User Interface in Collaborative Virtual Environments Application. Symmetry; 2019; 11, 476. [DOI: https://dx.doi.org/10.3390/sym11040476]

27. Chou, J.R. A psychometric user experience model based on fuzzy measure approaches. Adv. Eng. Inform.; 2018; 38, pp. 794-810. [DOI: https://dx.doi.org/10.1016/j.aei.2018.10.010]

28. Cross, N. Designerly ways of knowing: Design discipline versus design science. Des. Issues; 2001; 17, pp. 49-55. [DOI: https://dx.doi.org/10.1162/074793601750357196]

29. Chen, L.T. The Diffusion of Design Thinking: An Empirical Study in Greater China; National Tsing Hua University: Hsinchu City, Taiwan, 2017.

30. Wang, Y.L. The Method of Design Thinking into the Process of Developing the Scratch Curriculum of Environmental Education; National Tsing Hua University: Hsinchu City, Taiwan, 2017.

31. Brown, T. Change by Design: How Design Thinking Transforms Organizations and Inspires Innovation; Harper Business: New York, NY, USA, 2009.

32. Brown, T. Design Thinking. Harv. Bus. Rev.; 2008; 86, pp. 84-92.

33. Chan, J.T. The Study of Impact on Health Awareness Concept in Milling Agricultural Package Design Selection and Usability by Design Thinking Method; Ming Chuan University: Taipei City, Taiwan, 2017.

34. Hsiao, R.L. Organization of Thinking; 3rd ed. Global Views-Commonwealth Publishing: Taipei, Taiwan, 2011; ISBN 9789864790760

35. Qiu, R. Creative problem-solving strategies for science and technology teaching activities design-Wonderful ideas for “frames”. Life Sci. Educ. Mon.; 2008; 41, pp. 49-60.

36. Chen, L.A. Theory and Practice of Creative Thinking Teaching; 6th ed. Psychological Publishing: Taipei, Taiwan, 2006; ISBN 957702940X

37. Nagel, R.L.; Pappas, E.C.; Pierrakos, O. On a Vision to Educating Students in Sustainability and Design-The James Madison University School of Engineering Approach. Sustainability; 2012; 4, pp. 72-91. [DOI: https://dx.doi.org/10.3390/su4010072]

38. Meinel, C.; Leifer, L.; Plattner, H. Design Thinking; 1st ed. Springer: Berlin/Heidelberg, Germany, 2011; pp. 101-110. ISBN 9783642137570

39. Jobst, J.B.; Köppen, E.; Lindberg, T.; Moritz, J.; Rhinow, H.; Meinel, C. The Faith-Factor in Design Thinking: Creative Confidence through Education at the Design Thinking Schools Potsdam and Stanford. Design Thinking Research; Plattner, H.; Meinel, C.; Leifer, L. Springer International Publishing: Basel, Switzerland, 2012; pp. 35-46.

40. Carayannis, E.G. Encyclopedia of Creativity, Invention, Innovation and Entrepreneurship; Springer: New York, NY, USA, 2013; ISBN 9781461438588

41. Serrat, O. Design Thinking. Knowledge Solutions; Springer: Singapore, 2017; pp. 129-134.

42. Garbuio, M.; Lovallo, D. Design Thinking. The Palgrave Encyclopedia of Strategic Management; Augier, M.; Teece, D.J. Palgrave Macmillan: London, UK, 2018.

43. Tu, J.C.; Liu, L.X.; Wu, K.Y. Study on the Learning Effectiveness of Stanford Design Thinking in Integrated Design Education. Sustainability; 2018; 10, 2649. [DOI: https://dx.doi.org/10.3390/su10082649]