1. Introduction and Motivation
Teleconsultation has many benefits for both patients and healthcare practitioners. It makes healthcare more accessible to people, especially those living in remote areas where access to a doctor is difficult. It also eliminates the need for long travels and waiting times by allowing people to consult doctors from home [1]. The availability of doctors via teleconsultation ensures that patients can receive care even when their regular doctors are unavailable, at no additional cost. Moreover, teleconsultation is covered by health insurance, making it less expensive. Teleconsultation also saves time for healthcare practitioners, improves patient flow, and allows for more direct and efficient communication, all of which leads to improved care and continuity of treatment. Teleconsultation improves healthcare decision-making by enabling remote consultations with specialists, thereby enhancing decision quality, especially in rural or underserved areas [2]. It differs between asynchronous teleconsultation (where health information is transmitted without real-time participation) and synchronous teleconsultation (which involves real-time communication such as video or phone calls). It also promotes collaboration among healthcare practitioners, resulting in more accurate and comprehensive decisions.
Automatic Emotion Recognition (AER) brings several benefits to teleconsultation. It helps therapists notice small emotions that might be missed. As highlighted by Wolf (2015) [3], measuring facial expressions of emotion is a valuable tool for understanding emotional states and diagnosing mental health conditions, providing quantifiable indicators of emotional reactions and improving the accuracy of emotional assessments in clinical settings. AER gives consistent, clear data, reducing bias and making emotion tracking more reliable. It provides real-time feedback, allowing therapists to adjust quickly based on the patient’s feelings. AER also tracks emotions over time, helping therapists see changes and patterns. This leads to better decisions and more personalized care for the patient.
Facial emotion recognition (FER) uses deep learning techniques, especially Convolutional Neural Networks (CNNs), to detect emotions by analyzing facial expressions. The process has three main steps: first, the system detects and identifies the face in an image or video. Next, CNNs automatically extract key features like the eyes, eyebrows, and mouth. Finally, these features are classified into emotions like happiness, anger, or sadness. Deep learning makes CNNs effective at recognizing patterns in facial features. Ko (2018) [4] shows their success in FER, and [5] investigates CNNs, describing their components and key topologies from AlexNet to HRNet. Despite this progress, challenges like poor lighting and different facial expressions still exist. Figure 1 shows how FER works using CNNs. The process has two main steps. The first is feature extraction, where the input image goes through convolutional layers, followed by pooling layers to simplify the data while keeping important details. The second is classification, where the features are sent to fully connected layers, and the final layer uses softmax or sigmoid functions to classify the emotions.
In our paper, we propose the TheraSense system for mental health teleconsultation, featuring real-time facial emotion recognition based on CNNs. We explore how an affective computing (AC) system can be implemented using audiovisual sensors like cameras and microphones to detect emotions during teleconsultation. We also examine how AC and emotion detection can be applied to a teleconsultation platform for mental health, aiming to improve remote mental health care by providing real-time insights into patients’ emotions. Finally, we discuss how to evaluate the proposed system, considering the accuracy of the models, their performance, and user feedback regarding the functionalities or features of the system and user satisfaction.
This research adds an important feature to remote mental health care platforms by using real-time facial emotion recognition. Previous studies, like Kallipolitis et al. (2021) [6], focused on speech or limited visual cues to detect emotions. In contrast, this study uses CNNs to detect facial emotions during video calls. This helps therapists better understand patients’ emotions through nonverbal signals, which improves care. Chen and Li (2021) [7] focus on predicting teleconsultation demand, and Kanade et al. (2022) [8] work on stroke rehabilitation. Our study fills a gap by adding emotion detection to telehealth. Lu et al. (2021) [7] look at factors that affect patient satisfaction, but this study shows how emotion detection can improve satisfaction by making teleconsultations more personal. In short, this research advances telemental health care by making it more emotion-aware and improving remote care.
The research in this paper connects two projects: the SenseCare project [9], which explores the use of sensors in affective computing to enhance medical care, and the SMILE project [10], which seeks to support mental health in young people through an integrated approach to clinical decision-making and evidence-based interventions.
The remainder of this paper is organized as follows: Section 2 reviews related research on remote mental health care. Section 3 describes the conceptual design of TheraSense, including use cases and architecture. Section 4 outlines the implementation methodology. Section 5 presents the experimentation and evaluation of the proposed system. Lastly, Section 6 concludes the paper and highlights potential future directions.
2. Related Work
Teleconsultation in mental health has progressed with AI and machine learning, enabling real-time emotional analysis via voice, facial expressions, and text. These techniques enhance remote mental health care by increasing the precision of diagnoses and providing tailored treatment. For instance, Katsis et al. (2006) [11] described a telemedicine platform that uses biosensors and communication technology to analyze emotional and physiological states. It measures signals like heart rate and electrodermal activity in real time, providing feedback for clinical and remote monitoring. By analyzing these data, the platform offers a systematic way to assess and monitor patients’ mental and physical health from a distance. In the same way, Chan et al. (2014) [12] investigate mobile telemental health solutions, focusing on their usability and clinical outcomes. They recommend merging traditional and digital approaches to mental health care. Both research demonstrate how technology is influencing mental health assessments and therapies. Additionally, a study conducted by Tokuno et al. (2011) [13] showed its potential for assessing soldiers’ emotional states and improving mental health interventions. Similarly, Ramalho et al. (2022) [14] describe a better teleconsultation platform for depressive diseases that employs AI and machine learning to assess multimodal data. Further, Madanian et al. (2022) [15] used automatic speech emotion recognition to digitally adjust mental health treatment, demonstrating how ML (machine learning) can detect and interpret emotional states from speech, and highlights the capacity of SER (speech emotion recognition) as an initial step towards intelligent support systems in mental health. Our previous works [16] show that AI-based emotion recognition using facial expression analysis can improve short-term patient–provider communication. It helps healthcare practitioners make quick and accurate choices by combining real-time emotion recognition with simple visualizations, especially in telemedicine and therapy settings. Also, in earlier research [17], we investigated video-based emotion recognition using facial expressions and contactless vital signs. This method was designed for psychosomatic monitoring, demonstrating how these techniques can provide useful insights into emotional states and aid in mental health care. Furthermore, Di Carlo et al. (2021) [18] investigated the use of telepsychiatry and other technologies during the COVID-19 pandemic, with an emphasis on how these tools address emotional and mental health issues. They assessed how telepsychiatry provided emotional support and mental health care when in-person encounters were limited. Their study investigates patients’ emotional responses, engagement, and effects on mental health outcomes. It focuses on how telepsychiatry bridged physical hurdles and provided critical emotional support, assisting patients in dealing with worry, anxiety, and other emotional concerns throughout the epidemic. Moreover, Bashshur et al. (2016) [19] investigate how technology affects mental health by examining data from telemedicine interventions for mental illnesses. They examine a variety of research to determine the efficacy of telemedicine in treating emotional and psychological problems such as sadness and anxiety. Their findings demonstrate that telemedicine, particularly telepsychiatry, considerably improves mental health outcomes and is as effective as in-person care.
In summary, all of the studies discussed aim to improve telemedicine and remote mental health care. Our research expands on this by focusing on the application of deep learning, specifically Convolutional Neural Networks (CNNs), to detect emotions in patients’ facial expressions during remote teleconsultations. This approach contributes to real-time emotional insights, increasing the effectiveness of telemedicine.
3. Conceptual Design
This paper focuses on the TheraSense system, which enhances teleconsultation by integrating real-time emotion detection and AI-driven analysis to improve remote mental health care. During the Conceptual Modeling Phase, the User-Centered Design approach [20] is used to create use cases during the Conceptual Modeling Phase, ensuring the system meets user needs and expectations. After that, the Unified Modeling Language (UML) approach [21] is applied to organize and visualize the system.
The TheraSense System includes four use cases: UC 1.1, UC 1.2, UC 1.3, and UC 1.4. UC 1.3 focuses on developing a conceptual model and methods for using AC and emotion detection in mental health teleconsultation. These use cases are specifically designed to integrate AC (affective computing) and emotion detection for psychotherapy applications. Figure 2 shows the use context of the TheraSense System.
UC 1.3 addresses teleconsultation sessions with real-time emotion detection, providing insights into patients’ emotions to enhance therapy. Figure 3 illustrates the UC 1.3 diagram.
In UC 1.3 of the TheraSense System, the patient, therapist, and TheraSense System interact during scheduled teleconsultations. The system employs cameras, microphones, and Kinect devices to capture real-time emotions through facial expressions, voice, and body movements. The emotion data are shared with the therapist during the session and stored afterward, offering a summary to the therapist. This process provides the therapist with a clear understanding of the consultation’s progression and the patient’s wellbeing.
Figure 4 illustrates the TheraSense User Interfaces Diagram along with their functionalities.
The TheraSense User Interface Model has three main parts, designed to meet the needs of its users. The Patient Portal allows patients to log in, join teleconsultations, and sign consent forms electronically. The Therapist Dashboard is for therapists, enabling them to log in, manage their availability, schedule appointments, review patient details, manage consent forms, join teleconsultations scheduled for patients, and check emotion or session charts to adjust treatment plans. The Admin Console is for administrators, helping them manage the system, handle user accounts, and control access permissions. These interfaces work together to create a simple and effective experience for patients, therapists, and administrators.
4. Implementation
The TheraSense system uses a monolithic architecture, which makes development and deployment easy. All components operate on a single virtual machine on the Hagen University cloud, making maintenance easier. Node.js was chosen for the backend due to its effectiveness in dealing with asynchronous operations and scalability, and Express was chosen as the web framework to manage server-side functionality. Express manages routing for processes such as patient registration, appointment scheduling, consent management, teleconsultations, and administrative functions, ensuring that everything runs smoothly. Custom middleware is critical to the system’s functionality, performing activities like JSON parsing, logging requests, session management, and enforcing security standards to protect user data. This middleware provides support for the system’s functionality, such as appointment scheduling and teleconsultation. Figure 5 illustrates the implementation architecture of the TheraSense system.
The TheraSense system has two main parts: the Edge Layer and the Cloud Layer. The Edge Layer includes the user interfaces for patients, therapists, and administrators, allowing them to interact with the system. It also monitors the patient’s condition during teleconsultations in real-time. Data from microphones, cameras, and wearable sensors is processed using AI, deep learning (DL), and machine learning (ML) models in this layer. The processed data are then stored in the cloud. The Edge Layer also connects to a communication module that includes a WebSocket server for real-time teleconsultations, authentication, and data exchange. The therapist can view data and make decisions using their interface here. Currently, real-time monitoring of emotion is performed using the patient’s camera. Future integration is planned for monitoring patient vital signs and emotions using wearable sensors and microphones for further experimentation. The Cloud Layer stores all the data collected by the Edge Layer, such as user accounts, processed data, and appointments. It also stores backup copies of everything for security. In the cloud, the data are processed and organized in a database to ensure safe storage and easy access. The Cloud Layer also houses the AI, DL, and ML models used for further analysis and predictions, enhancing the system’s capabilities.
Hosting TheraSense on the Hagen Cloud ensures consistent performance, seamless scalability, and secure data management. Its robust architecture enables real-time emotion detection and CNN-based processing, facilitating the seamless analysis of large datasets. Centralized hosting promotes secure data visualization, allowing therapists to efficiently monitor emotional trends and patterns, ultimately improving user experiences.
TheraSense’s architecture enables direct peer-to-peer connections via WebRTC, allowing for real-time communication between a patient (Client B) and a therapist (Client A). The Node.js server works as a signaling gateway, transmitting crucial connection data such as offers and responses. Once the peer-to-peer link is established, video and audio streams are sent directly between the therapist and the patient, allowing for quick communication. This instantaneous sharing of information during teleconsultation sessions considerably improves efficiency and engagement, both of which are required for successful therapy and accurate monitoring of the patient’s emotions, as illustrated in Figure 6.
Further, the therapist dashboard displays the patient’s recognized emotion during the video call. It initiates video and audio streams for the therapist, establishing a peer connection with the local video and audio tracks. When the patient sends a video, it is shown in the remote video element on the therapist’s interface, allowing them to view both the patient’s video and emotional state in real time.
TheraSense is designed with emotion recognition functionality that accurately detects patients’ emotions from their facial expressions during teleconsultations. The following key technologies are used: (1) TensorFlow.js [22]: a JavaScript library for machine learning that allows you to create and run ML models directly in the web or in Node.js environments. (2) Face-api.js [23]: a JavaScript module based on TensorFlow.js that uses numerous CNNs for facial detection and emotion recognition, such as FaceExpressionNet, FaceLandmark68Net, FaceLandmark68TinyNet, FaceRecognitionNet, SsdMobilenetv1, TinyFaceDetector, MTCNN (Multi-Task Cascaded Convolutional Networks), and TinyYolov2. The face expression recognition model uses depthwise separable convolutions and densely connected blocks. The Perceptron model performs a linear transformation, but activation functions like Softmax or Sigmoid allow it to approximate non-linear functions. Figure 7 demonstrates the patient’s emotion recognition process in the browser during a teleconsultation. The TheraSense system analyzes video feeds in real time using deep learning techniques, specifically CNNs like TinyFaceDetector for face detection and FaceExpressionNet for facial expression recognition.
During the video call, the patient’s emotions are classified every 500 milliseconds into seven categories: happy, sad, angry, disgusted, fearful, neutral, and surprised. They are stored in a database to be visualized by the therapist at the end of the consultation. This approach works for both web and mobile platforms. The visualization of the patient’s detected emotions during a video meeting is implemented using a function that aggregates and categorizes emotional data into three categories: positive, negative, and neutral. The therapist can easily analyze these data because they are shown in a bar chart. The function retrieves emotion data linked with a particular consultation from the backend. Each observed emotion is then categorized using specified labels. In healthcare settings, “surprised” is considered a positive emotion as it encourages engagement and openness (Peng & Huang, 2020 [24]; Russ et al., 2013 [25]). Consequently, our formula includes “surprised” in the calculation of positive emotions. Thus, positive emotions consist of “happy” and “surprised”, while negative emotions include “sad”, “angry”, “fearful”, and “disgusted”. The “neutral” label represents only neutral emotions. The total for each category is calculated using the following formulas:
where Tpositive represents the total count of positive emotions, Tnegative represents the total count of negative emotions, and Tneutral represents the total count of neutral emotions. The emotion counts are displayed on a chart within a visible container on the therapist’s dashboard. Figure 8 illustrates an example of the output from the therapist’s dashboard.After the end of the video call consultation, the interactive chart provides valuable insights into the patient’s emotional state over time by categorizing emotions into three distinct categories: neutral, negative, and positive. It identifies emotional peaks, troubling moments, and times of equilibrium by monitoring time on the X-axis and emotional intensity on the Y-axis. While anger or sadness signify distress, other emotions like happiness or surprise may indicate satisfaction. Along with showing how emotions change in response to therapies, the chart can also show patterns of emotional recovery. Trends, peaks, and recovery periods can be analyzed to better understand emotional resilience and drive future patient treatment.
5. Experimentation and Evaluation
In the experimentation and evaluation phase, we will evaluate TheraSense using both quantitative and qualitative methods. The quantitative performance evaluation experiments use an AC system based on video emotion recognition. In this analysis, the quantitative evaluation of TheraSense’s performance is outlined using known approaches from previous research. Calingaert’s 1967 approach [26] emphasizes important factors such as response time, throughput, resource usage, dependability, scalability, and cost-effectiveness. These measurements serve to evaluate the system’s speed, efficiency, and adaptability. Furthermore, Lucas’ 1971 study [27] emphasizes continual monitoring of response time, output, and benchmarks to quickly identify and correct problems, ensuring system reliability and performance. The qualitative evaluation will include an online survey analysis to evaluate the TheraSense system, based on Punter et al.’s (2003) publication [28]. It highlights essential steps in survey design, such as providing simple questions, pretesting, and offering incentives to enhance response rates and data accuracy. This method enables TheraSense to efficiently gather structured feedback on usability and user satisfaction. The third evaluation of TheraSense used a qualitative heuristic evaluation, following Nielsen’s (1992) heuristic evaluation [29]. Heuristic evaluation is a method of discovering usability flaws that involves having a few experts review an interface using heuristics or known usability criteria. Every specialist studies the user interface independently, looking for issues like uneven design, a lack of feedback, or techniques that avoid user errors.
5.1. Quantitative Performance Evaluation of Video-Based Emotion Recognition System
Face detection is one of the most critical elements in emotion recognition systems. Emotions are often represented by facial expressions, which can be detected and investigated in video frames, so face detection and emotion recognition models are related. The quality of the face detection model is important because it ensures that the emotion recognition system examines the relevant parts of the video, which directly affects the accuracy of the emotion detection system. Without good face recognition, the emotion recognition process will be affected.
Even though audio-based recognition of emotions can work well in controlled settings, it becomes less accurate in dynamic and real-world scenarios when there is background noise or fluctuating audio quality. In contrast, video-based emotion detection—specifically, facial expression monitoring—has shown greater adaptability in a variety of scenarios, such as dynamic environments and user movement. The framework developed by Khan et al. (2018) [30] emphasizes the significance of integrating data analysis with practical testing to ensure dependable system performance, and the performance evaluation approach is similar to this framework. This assessment evaluates the accuracy and usefulness of the system in real-world situations.
This link delivered a script to assess the effectiveness of three distinct face detection models: Tiny Face Detector, MTCNN, and SSD MobileNet: URL
After the user uploads an image and chooses a model from a dropdown menu, face-api.js is used by the system to identify faces. The server (/models) loads the models. On a canvas overlay, bounding boxes are used to highlight the faces that have been detected.
Common face detection models with different default hardware requirements include SSD MobileNet, MTCNN, and Tiny Face Detector. Designed for mobile and embedded devices, SSD MobileNet can run on a CPU but typically uses a GPU for better performance. Usually operating on a CPU, MTCNN is a more CPU-intensive model that can benefit from GPU acceleration for quicker inference. Lightweight and built to run efficiently on a CPU, Tiny Face Detector can also run on a GPU if one is available, enabling quicker processing. The script also calculates and displays the inference time, which is the time required to process the image and detect faces, as well as the number of detected faces. The inference time is calculated by recording the start and end times of the face detection process. When the user selects a new model from the menu, the script reloads the model to ensure accurate detection. This setup enables real-time evaluation of the models’ performance by comparing detection accuracy and processing speed based on uploaded images.
The experiment was run on a MacBook Air with an Apple M2 chip, 24 GB of RAM, and macOS 14.1.1 (23B81), One Apple Park Way, Cupertino CA 95014, USA. This configuration was used to compare the real-time face detection performance of three models: SSD MobileNet, MTCNN, and Tiny Face Detector. The investigation focused on key variables such as inference time, accuracy, and the number of faces detected per image. The results are summarized in Table 1. The Image column represents the image being evaluated, and each image has a known number of faces. The images utilized in this study were obtained from a variety of free sources, as well as publicly available Google images. They include files in several formats such as PNG, JPG, JPEG, and WEBP, ensuring a wide variety of image types. Images used for evaluation, image1, image2, image3, image4, and image5, are available at the following URL:
The Number of Faces in Image column represents the total number of faces present in the image, which serves as the ground truth for evaluating the models’ performance. This value is determined prior to testing and reflects the actual count of faces present in the image. The Model Used column specifies the face detection model applied to analyze the image. The table includes the three selected models: SSD MobileNet, MTCNN, and Tiny Face Detector. Each of these models performs differently in terms of accuracy and processing speed, making them suitable for a variety of application scenarios. The Inference Time (ms) column indicates the time required for a model, in milliseconds, to process the image and produce detection results. A lower inference time indicates a faster model, which is particularly beneficial for real-time applications requiring quick detection. The confidence score for each face detection is determined using a probabilistic method. This score is calculated based on the output of a detection model (such as SSD, MTCNN, etc.), which assigns a probability to each face detection. In our script, the parameter “detection.score” represents the model’s confidence that a specific detection is correct, and it ranges from 0 to 1, with a number closer to 1 indicating greater confidence. The script then uses “.toFixed(2)” to format this score to two decimal places, which it presents in the results section for each detected face. Finally, the Accuracy column indicates how effectively the model detects faces in an image. It shows the ratio of accurately recognized faces to the total number of actual faces in the image. Higher accuracy indicates a better detecting model. The formula for calculating it is
where Faces Detected is the count from the model’s detection array and Number of Faces in Image is the ground truth count, which can be determined manually or in advance. When the accuracy is 100%, it means the model performed perfectly on that image. It detected all of the faces correctly.The results indicate that while MTCNN delivered the best accuracy, achieving 100% in several images (Image 1, 2, and 4), it came with a significant choice in processing time. For example, MTCNN took about 3 s for smaller images and nearly 14 s for larger images, which is quite slow. On the other hand, SSD MobileNet, although quicker (less than 0.5 s), displayed a drop in accuracy when processing larger images. The Tiny Face Detector, being the fastest with an inference time of about 200 milliseconds, had the lowest accuracy, particularly in images with more faces, and it even failed to detect faces in Image 5. In summary, MTCNN provides the best accuracy but at the cost of slower performance, while SSD MobileNet offers a good balance of speed and accuracy, and Tiny Face Detector, though fast, struggles with accuracy, especially in more complex images.
A comprehensive comparison of the system’s performance based on various neural network models for real-time face detection is shown in Figure 9, taking into account inference time, accuracy, and hardware requirements.
The chart compares three face detection models: SSD Mobilenet V1, Tiny Face Detector, and MTCNN. MTCNN is the most accurate but also the slowest. Tiny Face Detector is the fastest but has the lowest accuracy. SSD Mobilenet V1 provides a middle ground with good accuracy and reasonable speed. The chart highlights the differences between accuracy and speed in these models. For real-time tasks, SSD MobileNet V1 provides the perfect speed-accuracy ratio, outperforming GPUs. Tiny Face Detector is best suited for CPUs with optional GPU capability; however, it is less reliable. High accuracy is best achieved with MTCNN, which is slower, particularly on CPUs, and benefits from GPU acceleration. The accuracy of facial expression-based emotion recognition was also found to decline with decreasing illumination levels at the experiment site. Thus, recognizing facial expressions in videos with high accuracy and fast inference time is the primary challenge.
5.2. Qualitative Heuristic Evaluation of Emotion Recognition in TheraSense User Interface
The qualitative heuristic evaluation of TheraSense will be used to focus on identifying potential usability issues and areas for improvement in emotion recognition and detection using audiovisual sensors with CNN-based accuracy. These potential usability issues and areas for improvement will be identified through expert analysis.
The detection of a patient’s emotions on the therapist’s dashboard requires an excellent network connection between the therapist and the patient. In addition, the therapist’s device (PC, laptop, tablet, or smartphone) should have at least 6 GB of memory to handle the data processing requirements of real-time emotion detection. A dedicated GPU is also required to effectively run CNNs models and maintain smooth performance. Furthermore, therapists can miss essential emotional indicators if negative feelings are noticed throughout the session. The system requires an automated alert to tell therapists when negative emotions are recognized and exceed positive emotions at the end of the session on the therapist dashboard.
The video call functionality of TheraSense was tested on a variety of devices, including a MacBook Air with an Apple M2 and 24 GB of RAM, as well as two smartphones: a Redmi 9T and a Samsung model. This setup enabled an evaluation of compatibility and performance across a variety of hardware configurations. Other suggestions for improving the user interface for both therapists and patients include correcting the absence of a clearly labeled “End Call” button during video calls, which can cause confusion about how to terminate a call. Adding a visible and immediately accessible “End Call” button would improve the overall user experience. Furthermore, users may not receive confirmation notifications following account creation, leaving them unsure of the next steps. Therefore, a notification system that verifies account creation and sends a welcome message with explicit directions on the next steps should be implemented. Additionally, if therapists do not actively monitor sessions, they may miss out on crucial emotional information. By using an automated alerting system, therapists may be sure they are aware of critical indications when negative emotions are recognized at the end of a session. In order to improve navigation and streamline the design, user testing is necessary if the user interface appears packed or unclear. There should be a feedback system in place that makes it easy for users to report problems. Table 2 presents a summary of the qualitative heuristic evaluation of the TheraSense platform, focusing on usability challenges and recommended improvements.
5.3. Quantitative Evaluation of the TheraSense System Through Survey Analysis
The goal of the TheraSense Prototype Evaluation Survey was to collect feedback from technical experts and mental health practitioners in order to assess the system’s efficacy. The survey was open to participants from a variety of professions, including psychologists, psychiatrists, counselors, social workers, and even computer science academics. The survey was made available to a diverse group of participants, including professionals such as psychologists, psychiatrists, counselors, social workers, and academics in computer science. It inquired about the ease of use of the interface, the intuitiveness of accessing emotional data, and the ability to perform essential tasks without difficulty. Participants rated these aspects on a 1–5 scale, where 1 represented “Very Difficult” or “Not Intuitive”, and 5 represented “Very Easy” or “Very Intuitive. The survey link was sent along with the credentials to access the system as a patient, therapist, or admin. Table 3 summarizes the survey ratings of TheraSense identified during the evaluation.
The survey results highlight the experience with the TheraSense system. Most users had 1–5 years of experience in mental health. All participants (100%) found the interface easy to navigate (rated 5.0) and agreed tasks were easy to complete (5.0). The controls for accessing patient emotional data were considered intuitive, with users giving high ratings. Tasks like viewing emotional states were completed without difficulty. The ability to see patients’ emotional states had varying impacts: the majority of respondents (75%) reported a positive change to some degree, with 25% finding it somewhat changed their interactions. Another 25% found it neutral, while 50% found it significantly changed the way they interacted with patients. The system was considered effective in enhancing teleconsultations. Key features that were appreciated included real-time emotion detection, teleconsultation functionality, secure patient data tracking, and the appointment scheduler. Suggestions included adding continuous monitoring of vital signs, a single login per group, an “End Call” button, and better alerts for abnormal vital signs. Some technical issues were noted, such as the “Visualize Emotions” button not working.
Following those adjustments, the bug related to the visualization of the emotion button was fixed. The call management button and mute features were added. Future work will focus on integrating vital sign monitoring (e.g., heart rate, respiration rate), real-time notifications, and an online prescription feature.
6. Conclusions and Future Work
To summarize, the TheraSense system was successfully constructed using a monolithic architecture that includes both frontend and backend components. It efficiently manages patient data, therapist data, scheduled consultations, and administrative data, ensuring a consistent experience for patients, therapists, and administrators. The system enables real-time communication and secure data management through the use of technologies such as Node.js, MongoDB, and WebRTC. Future enhancements could concentrate on developing the system and introducing additional features depending on user experience. The interactive chart provides an engaging way to analyze patients’ emotions after a video consultation. It categorizes emotions as neutral, negative, or positive, tracks changes over time, and highlights emotional peaks, distress indicators, and periods of recovery. This helps understand emotional resilience and will guide future treatment strategies.
TheraSense uses CNN-based models to accurately detect emotions in real time using audio and visual sensors. Using these models, therapists can successfully track their patients’ emotional states during sessions. CNNs enhance system performance with dedicated GPU support and powerful processing capabilities, allowing for accurate emotion recognition even in dynamic consultation settings. Furthermore, the evaluation of video-based emotion recognition systems found that SSD MobileNet V1 offers the best balance of speed and accuracy for real-time tasks, while MTCNN provides the highest accuracy but is slower. Additionally, low-light conditions decrease the accuracy of facial expression-based emotion recognition.
Additionally, the TheraSense usability evaluation showed that users found the interface easy to use and navigate. Most tasks, like accessing emotional data, were simple. This means the platform is user-friendly and useful for remote mental health care. Both patients and therapists can use it easily.
The limitations of the TheraSense system are mainly related to hardware and network conditions. It requires a stable internet connection for video streaming, good lighting for accurate emotion detection, and devices with sufficient memory and power to process data effectively.
Future Research Trends
In future work, TheraSense will be benchmarked against existing tools, such as i-Motion and Affectiva. This will help identify its strengths and areas for improvement. The focus will be on enhancing the functionality of the TheraSense platform. Planned updates include continuous vital sign monitoring via wearable sensors or camera, real-time notifications, and the ability to provide online prescriptions. These improvements will offer a more comprehensive approach to remote mental health care, enabling better tracking and faster responses. Future research may also explore combining facial expressions with voice data from microphones and text analysis, including chatbots, to improve emotion recognition. Additionally, AI tools could assist therapists in making more informed decisions during consultations. These innovations aim to make mental health care more efficient, adaptable, and focused on the needs of both patients and therapists. The next phase of the project will include testing teleconsultation therapies with actual patients and therapists, with the goal of assessing their real-world effectiveness and feasibility.
Conceptualization, H.H.; Methodology, H.H.; Validation, B.V. and M.H.; Investigation, M.H.; Writing—original draft, H.H.; Supervision, B.V. and M.H. All authors have read and agreed to the published version of the manuscript.
The original contributions presented in this study are included in the article. Further inquiries can be directed to the corresponding authors.
The authors declare no conflicts of interest.
Footnotes
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.
Enlarge this image.Figure 7. Patient emotion recognition process in the browser during teleconsultation.
Inference time, confidence score, and accuracy of real-time face detection: SSD MobileNet, MTCNN, Tiny Face.
| Image | Number of Faces in Image | Model Used | Inference Time (ms) | Confidence Score | Accuracy |
|---|---|---|---|---|---|
| image1 | 1 | SSD MobileNet | 349.90 ms | Face 1: 0.99 | 100% |
| image1 | 1 | MTCNN | 2935.90 ms | Face 1: 1.00 | 100% |
| image1 | 1 | Tiny Face Detector | 203.00 ms | Face 1: 0.91 | 100% |
| image2 | 4 | SSD MobileNet | 336.20 ms | Face 1: 0.99 Face 2: 0.99 Face 3: 0.99 Face 4: 0.99 | 100% |
| image2 | 4 | MTCNN | 3563.20 ms | Face 1: 1.00 Face 2: 1.00 Face 3: 1.00 Face 4: 1.00 | 100% |
| image2 | 4 | Tiny Face Detector | 206.90 ms | Face 1: 0.81 Face 2: 0.78 Face 3: 0.73 | 75% |
| image3 | 4 | SSD MobileNet | 329.80 ms | Face 1: 1.00 Face 2: 0.99 Face 3: 0.99 Face 4: 0.98 | 100% |
| image3 | 4 | MTCNN | 17,746.80 ms | Face 1: 1.00 Face 2: 1.00 Face 3: 1.00 | 75% |
| image3 | 4 | Tiny Face Detector | 205.20 ms | Face 1: 0.99 Face 2: 0.89 Face 3: 0.80 Face 4: 0.66 | 100% |
| image4 | 18 | SSD MobileNet | 399.10 ms | Face 1: 0.99 Face 2: 0.99 Face 3: 0.98 Face 4: 0.98 Face 5: 0.97 Face 6: 0.96 Face 7: 0.96 Face 8: 0.96 Face 9: 0.93 Face 10: 0.92 Face 11: 0.91 Face 12: 0.91 Face 13: 0.90 Face 14: 0.82 Face 15: 0.82 Face 16: 0.80 Face 17: 0.59 | 94.44% |
| image4 | 18 | MTCNN | 13,994.60 ms | Face 1: 1.00 Face 2: 1.00 Face 3: 1.00 Face 4: 1.00 Face 5: 1.00 Face 6: 1.00 Face 7: 1.00 Face 8: 1.00 Face 9: 1.00 Face 10: 1.00 Face 11: 1.00 Face 12: 1.00 Face 13: 1.00 Face 14: 1.00 Face 15: 0.97 Face 16: 0.97 Face 17: 0.95 Face 18: 0.85 | 100% |
| image4 | 18 | Tiny Face Detector | 207.30 ms | Face 1: 0.80 Face 2: 0.77 Face 3: 0.74 Face 4: 0.68 Face 5: 0.67 Face 6: 0.61 Face 7: 0.61 | 38.88% |
| image5 | 32 | SSD MobileNet | 383.40 ms | Face 1: 1.00 Face 2: 1.00 Face 3: 1.00 Face 4: 0.99 Face 5: 0.99 Face 6: 0.99 Face 7: 0.99 Face 8: 0.99 Face 9: 0.99 Face 10: 0.99 Face 11: 0.99 Face 12: 0.98 Face 13: 0.98 Face 14: 0.98 Face 15: 0.98 Face 16: 0.97 Face 17: 0.96 Face 18: 0.95 Face 19: 0.94 Face 20: 0.93 Face 21: 0.93 Face 22: 0.93 Face 23: 0.93 Face 24: 0.92 Face 25: 0.91 Face 26: 0.90 Face 27: 0.89 Face 28: 0.83 Face 29: 0.80 Face 30: 0.76 Face 31: 0.76 | 96.87% |
| image5 | 32 | MTCNN | 18,542.90 ms | Face 1: 1.00 Face 2: 1.00 Face 3: 1.00 Face 4: 1.00 Face 5: 1.00 Face 6: 1.00 Face 7: 1.00 Face 8: 1.00 Face 9: 1.00 Face 10: 1.00 Face 11: 1.00 Face 12: 1.00 Face 13: 1.00 Face 14: 1.00 Face 15: 1.00 Face 16: 1.00 Face 17: 1.00 Face 18: 1.00 Face 19: 1.00 Face 20: 1.00 Face 21: 1.00 Face 22: 1.00 Face 23: 1.00 Face 24: 1.00 Face 25: 1.00 Face 26: 1.00 Face 27: 1.00 Face 28: 1.00 Face 29: 0.99 Face 30: 0.99 Face 31: 0.98 | 96.87% |
| image5 | 32 | Tiny Face Detector | 197.60 ms | No faces detected | 0% |
Usability issues and improvements for TheraSense.
| Usability Concern | Description | Recommended Improvement |
|---|---|---|
| Network and Device Requirements | Real-time emotion detection on the therapist’s dashboard requires a high-quality network, at least 6 GB of memory, and a dedicated GPU on the therapist’s device. | Ensure a stable, high-speed network and provide the therapist’s device with 6 GB of memory and a dedicated GPU for smooth real-time emotion detection. |
| Emotion Detection Alert System for Therapists | Therapists may miss important emotional cues if negative emotions go undetected. An automated alert should notify therapists when negative emotions surpass positive ones by the end of the session. | Implement an automated alert system to notify therapists when negative emotions exceed positive ones at the end of a session. |
| Lack of “End Call” Button | Absence of a clearly designated “End Call” button during video calls can cause uncertainty on how to terminate a call. | Implement a visible and easily accessible “End Call” button to improve user experience. |
| No Notifications After Account Creation | Users may not receive messages after creating an account, leading to confusion about the next steps. | Introduce a notification system that confirms account creation with a welcome message and instructions. |
| Cluttered or Unintuitive User Interface | The user interface may be confusing and difficult to navigate. | Conduct user testing to simplify the design and enhance navigation for better usability. |
| Lack of Feedback Mechanism | Users might find it difficult to report problems or provide feedback. | Implement a feedback mechanism that allows users to easily report issues or suggest improvements. |
| Emotion Detection Alert System for Therapists | The system needs an automated alert to notify therapists when negative emotions are detected and exceed positive emotions at the end of the session. | Set up an automated alerting system to notify therapists when negative emotions are detected and exceed positive emotions at the end of a session. |
TheraSense survey ratings.
| Question | Scale | Average Rating |
|---|---|---|
| Ease of Navigating the Interface | 1 (Very Difficult)–5 (Very Easy) | 100% |
| Intuitiveness of Controls | 1 (Not Intuitive)–5 (Very Intuitive) | 66.7% |
| Task Completion (e.g., Viewing Emotional States) | 1 (Strongly Disagree)–5 (Strongly Agree) | 100% |
| Impact of Seeing Patients’ Emotional States | 1 (Not at All)–5 (Significantly) | 66.7% |
| Effectiveness of TheraSense in Teleconsultations | 1 (Not Effective)–5 (Very Effective) | 66.7% |
References
1. Opoku, D.; Scott, P.; Quentin, W. Healthcare professionals’ perceptions of the benefits and challenges of a teleconsultation service in the Amansie-West district of Ghana. Telemed. e-Health; 2015; 21, pp. 748-755. [DOI: https://dx.doi.org/10.1089/tmj.2014.0210] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/25942579]
2. Deldar, K.; Bahaadinbeigy, K.; Tara, S.M. Teleconsultation and clinical decision making: A systematic review. Acta Inform. Medica; 2016; 24, 286. [DOI: https://dx.doi.org/10.5455/aim.2016.24.286-292] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27708494]
3. Wolf, K. Measuring facial expression of emotion. Dialogues Clin. Neurosci.; 2015; 17, pp. 457-462. [DOI: https://dx.doi.org/10.31887/DCNS.2015.17.4/kwolf] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26869846]
4. Ko, B.C. A brief review of facial emotion recognition based on visual information. Sensors; 2018; 18, 401. [DOI: https://dx.doi.org/10.3390/s18020401] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29385749]
5. Alzubaidi, L.; Zhang, J.; Humaidi, A.J.; Al-Dujaili, A.; Duan, Y.; Al-Shamma, O.; Santamaría, J.; Fadhel, M.A.; Al-Amidie, M.; Farhan, L. Review of deep learning: Concepts, CNN architectures, challenges, applications, future directions. J. Big Data; 2021; 8, pp. 1-74. [DOI: https://dx.doi.org/10.1186/s40537-021-00444-8] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33816053]
6. Kallipolitis, A.; Galliakis, M.; Menychtas, A.; Maglogiannis, I. Speech based affective analysis of patients embedded in telemedicine platforms. Proceedings of the 2021 43rd Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC); Guadalajara, Mexico, 1–5 November 2021; pp. 1857-1860.
7. Shi, J.; Ye, M.; Chen, H.; Lu, Y.; Tan, Z.; Fan, Z.; Zhao, J. Enhancing efficiency and capacity of telehealth services with intelligent triage: A bidirectional LSTM neural network model employing character embedding. BMC Med. Inform. Decis. Mak.; 2023; 23, 269. [DOI: https://dx.doi.org/10.1186/s12911-023-02367-1] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/37990204]
8. Kanade, A.; Sharma, M.; Muniyandi, M. Tele-EvalNet: A low-cost, teleconsultation system for home based rehabilitation of stroke survivors using multiscale CNN-ConvLSTM architecture. European Conference on Computer Vision; Springer: Cham, Switzerland, 2022; pp. 738-750.
9. Engel, F.; Bond, R.; Keary, A.; Mulvenna, M.; Walsh, P.; Zheng, H.; Wang, H.; Kowohl, U.; Hemmje, M. Sensecare: Towards an experimental platform for home-based, visualisation of emotional states of people with dementia. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Bornschlegl, M.X.; Engel, F.C.; Bond, R.; Hemmje, M.L. Springer International Publishing: Cham, Switzerland, 2016; pp. 63-74. [DOI: https://dx.doi.org/10.1007/978-3-319-50070-6_5]
10. SMILE. A Platform for Strengthening the Mental Health of Young People. Available online: https://www.horizonsmile.eu/project/ (accessed on 12 May 2024).
11. Katsis, C.D.; Ganiatsas, G.; Fotiadis, D.I. An integrated telemedicine platform for the assessment of affective physiological states. Diagn. Pathol.; 2006; 1, 16. [DOI: https://dx.doi.org/10.1186/1746-1596-1-16] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/16879757]
12. Chan, S.R.; Torous, J.; Hinton, L.; Yellowlees, P. Mobile tele-mental health: Increasing applications and a move to hybrid models of care. Healthcare; 2014; 2, pp. 220-233. [DOI: https://dx.doi.org/10.3390/healthcare2020220] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27429272]
13. Tokuno, S.; Tsumatori, G.; Shono, S.; Takei, E.; Yamamoto, T.; Suzuki, G.; Mituyoshi, S.; Shimura, M. Usage of emotion recognition in military health care. Proceedings of the 2011 Defense Science Research Conference and Expo (DSR); Singapore, 3–5 August 2011; pp. 1-5.
14. Ramalho, D.; Constantino, P.; Da Silva, H.P.; Constante, M.; Sanches, J. An augmented teleconsultation platform for depressive disorders. IEEE Access; 2022; 10, pp. 130563-130571. [DOI: https://dx.doi.org/10.1109/ACCESS.2022.3228324]
15. Madanian, S.; Parry, D.; Adeleye, O.; Poellabauer, C.; Mirza, F.; Mathew, S.; Schneider, S. Automatic speech emotion recognition using machine learning: Digital transformation of mental health. Proceedings of the Annual Pacific Asia Conference on Information Systems (PACIS); Sydney, Australia, 5–9 July 2022.
16. Hadjara, H.; Vua, B.; Maiera, D.; Mayerb, G.; Mc Kevittc, P.; Hemmje, M. Video-based emotion detection analyzing facial expressions and contactless vital signs for psychosomatic monitoring. Proceedings of the Collaborative European Research Conference; Cork, Ireland, 9-10 September 2021; 6473.
17. Hadjar, H.; Lange, J.; Vu, B.; Engel, F.; Mayer, G.; Mc Kevitt, P.; Hemmje, M.L. Video-based automated emotional monitoring in mental health care supported by a generic patient data management system. Proceedings of the PSYCHOBIT; Naples, Italy, 28–29 September 2020.
18. Di Carlo, F.; Sociali, A.; Picutti, E.; Pettorruso, M.; Vellante, F.; Verrastro, V.; Martinotti, G.; di Giannantonio, M. Telepsychiatry and other cutting-edge technologies in COVID-19 pandemic: Bridging the distance in mental health assistance. Int. J. Clin. Pract.; 2021; 75, [DOI: https://dx.doi.org/10.1111/ijcp.13716] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32946641]
19. Bashshur, R.L.; Shannon, G.W.; Bashshur, N.; Yellowlees, P.M. The empirical evidence for telemedicine interventions in mental disorders. Telemed. e-Health; 2016; 22, pp. 87-113. [DOI: https://dx.doi.org/10.1089/tmj.2015.0206] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26624248]
20. Pea, R.D. User Centered System Design: New Perspectives on Human-Computer Interaction. J. Educ. Comput. Res.; 1987; 3, pp. 129-134.
21. Muller, P.A. Modélisation Objet Avec UML; Eyrolles: Paris, France, 2000; 514.Available online: ftp://hackbbs.org/Dev/Methodologie/UML/coursUML.pdf (accessed on 15 January 2025).
22. TensorFlow.js|Le Machine Learning Pour les Développeurs JavaScript. Available online: https://www.tensorflow.org/js (accessed on 26 July 2021).
23. GitHub—Justadudewhohacks/Face-api.js: JavaScript API for Face Detection and Face Recognition in the Browser and Nodejs with Tensorflow.js. Available online: https://github.com/justadudewhohacks/face-api.js/ (accessed on 26 July 2021).
24. Peng, W.; Huang, Q. An examination of surprise and emotions in the processing of anecdotal evidence. Health Commun.; 2019; 35, pp. 766-777. [DOI: https://dx.doi.org/10.1080/10410236.2019.1587813] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30871378]
25. Russ, L.R.; Phillips, J.; Brzozowicz, K.; Chafetz, L.A.; Plsek, P.E.; Blackmore, C.C.; Kaplan, G.S. Experience-based design for integrating the patient care experience into healthcare improvement: Identifying a set of reliable emotion words. Healthcare; 2013; 1, pp. 91-99. [DOI: https://dx.doi.org/10.1016/j.hjdsi.2013.07.004] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/26249779]
26. Calingaert, P. System performance evaluation: Survey and appraisal. Commun. ACM; 1967; 10, pp. 12-18. [DOI: https://dx.doi.org/10.1145/363018.363040]
27. Lucas, H., Jr. Performance evaluation and monitoring. ACM Comput. Surv.; 1971; 3, pp. 79-91. [DOI: https://dx.doi.org/10.1145/356589.356590]
28. Punter, T.; Ciolkowski, M.; Freimut, B.; John, I. Conducting on-line surveys in software engineering. Proceedings of the 2003 International Symposium on Empirical Software Engineering, 2003 (ISESE); Rome, Italy, 30 September–1 October 2003; pp. 80-88.
29. Nielsen, J. Finding usability problems through heuristic evaluation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; Monterey, CA, USA, 3–7 May 1992; pp. 373-380.
30. Khan, S.A.; Kusi-Sarpong, S.; Arhin, F.K.; Kusi-Sarpong, H. Supplier sustainability performance evaluation and selection: A framework and methodology. J. Clean. Prod.; 2018; 205, pp. 964-979. [DOI: https://dx.doi.org/10.1016/j.jclepro.2018.09.144]
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Vu, Binh 2 ; Hemmje, Matthias 1 1 Faculty of Mathematics and Computer Science, University of Hagen, 58097 Hagen, Germany;
2 Applied Data Science and Analytics, SRH University Heidelberg, 69123 Heidelberg, Germany