Content area
Healthcare organizations must evolve and adopt current evidence-based practices while maintaining care quality and operational effectiveness. Human-computer interaction is essential to enhancing clinical trial management processes and ensuring patient safety by optimizing how healthcare providers engage with critical information systems. However, technology transitions present challenges that require effective training and evaluation to ensure smooth adoption and usability. To strengthen clinical research capabilities, a comprehensive workflow and process mapping strategies was implemented to develop a novel custom-built electronic medical record application, PowerTrials. Understanding how healthcare team members interact with the PowerTrials interface during training is crucial for identifying potential shortcomings and optimizing future training designs. This study aimed to improve PowerTrials training by analyzing staff perceptions, performance, and user experience through biometric data. Two pre-recorded videos on clinical trial information and PowerTrials interface navigation were presented to 17 and 27 clinical participants, respectively, from a rural hospital in the western United States. Eye-tracking and facial expressions, along with quiz scores were collected to evaluate the PowerTrials training sessions. On average, participants directed more visual attention to key text (2,707 fixations) compared to random images (390 fixations) and focused more on central screen content (1,995 fixations) than on side content (512 fixations). Negative emotions were noted more frequently (38.6%) than positive ones (0.4%) based on emotional intensity. Pop-up windows and audible explanations effectively captured participant attention. Quiz results showed higher scores for the PowerTrials interface training (91.3/100) than for the clinical trial information training (85/100). Future research will evaluate user interaction with PowerTrials following training to further refine the interface.
Abstract
Healthcare organizations must evolve and adopt current evidence-based practices while maintaining care quality and operational effectiveness. Human-computer interaction is essential to enhancing clinical trial management processes and ensuring patient safety by optimizing how healthcare providers engage with critical information systems. However, technology transitions present challenges that require effective training and evaluation to ensure smooth adoption and usability. To strengthen clinical research capabilities, a comprehensive workflow and process mapping strategies was implemented to develop a novel custom-built electronic medical record application, PowerTrials. Understanding how healthcare team members interact with the PowerTrials interface during training is crucial for identifying potential shortcomings and optimizing future training designs. This study aimed to improve PowerTrials training by analyzing staff perceptions, performance, and user experience through biometric data. Two pre-recorded videos on clinical trial information and PowerTrials interface navigation were presented to 17 and 27 clinical participants, respectively, from a rural hospital in the western United States. Eye-tracking and facial expressions, along with quiz scores were collected to evaluate the PowerTrials training sessions. On average, participants directed more visual attention to key text (2,707 fixations) compared to random images (390 fixations) and focused more on central screen content (1,995 fixations) than on side content (512 fixations). Negative emotions were noted more frequently (38.6%) than positive ones (0.4%) based on emotional intensity. Pop-up windows and audible explanations effectively captured participant attention. Quiz results showed higher scores for the PowerTrials interface training (91.3/100) than for the clinical trial information training (85/100). Future research will evaluate user interaction with PowerTrials following training to further refine the interface.
Keywords
Healthcare System Design, EHR, Human Factors, Human Computer Interaction
1. Introduction
Healthcare organizations, like other large private and public organizations, must continue to evolve and change to keep current with evidence-based practices but also to maintain care quality, operational effectiveness, and efficiency [1]. Private industry has a long-standing history of proactively addressing organizational readiness to change [2], organizational resilience and change management. Yet these core dimensions can be implemented in healthcare organizations that enable the use of best practices and lessons learned for a broader impact. Specifically, hospitals can proactively detect those items that are likely to impede successful change using practical managerial tools to implement a transformation, planned or sudden, that enhances organizational capability [3, 4]. Any organizational change, especially in organizations with high-risk, is a challenge for leaders, managers and staff. Many factors hinder the process but especially the decoupling phenomena, which is a discrepancy between theoretical strategic decisions and concrete operational change [5]. Private industry and large public organizations have documented a framework and even a healthcare specific road map for hospital managers [6, 7]. Based on these resources and clustered domains that lead to successful organization change [8, 9], the purpose of this study was to assess the impact of change on clinical staff members that support clinical trials and those that provide inpatient and outpatient care. It was hypothesized that the clinical staff's readiness to change and attitudes would impact the acceptance of the new clinical trial management program. This research examined clinical research enrollment practices, behaviors, and lived experiences in the Rocky Mountain West region, which is largely rural or frontier designated.
2. Methodology
The study was deemed exempt by the health care system Privacy and Exemption Committee (Protocol #23.005) and approved by the partner university Institutional Review Board (Protocol #2023-604). The organization is a multi-specialty group practice with a 336-bed community teaching hospital and Level I trauma center serving four states. The health care system has a diverse range of service lines including surgical, inpatient and outpatient services. The university partner and its systems engineering department enabled staff surveys and data to be collected external to the health care organization for anonymity and confidentiality and conducted statistical consultation and analyses. The university is the largest in the four-state region with a student enrollment of 16,978, 570 tenurable faculty and $230 million in research expenditures. To explore integration of a new Electronic Health Records (EHR) application specific to clinical trial management, the team used facial expression analysis and eye tracking to assess clinical decision-making skills, and to improve training effectiveness. Sticky (Tobii AB, Stockholm, Sweden) was used to collect competency-based assessments (quizzes), eye tracking fixations and saccades, and facial expressions for two organization-based training and awareness modules. Eye tracking was collected from each subject when using the EHR. Web-based software was used to collect eye tracking data. Emotion data was collected from each subject while using the EHR. Web-based software in the laptop used during data collection period videotaped the entire process. A webcam-based research tool, Sticky, integrated into a Qualtrics survey was used to evaluate the user experience in the PowerTrials application. This nurse-led research study evaluated the novel electronic medical record application, PowerTrials, to monitor patients in clinical trials. Evaluating this application's user interface based on staff perceptions, performance and user experience, used screen-based eye tracking, performance metrics, and facial expressions to deploy the novel application in a more user-centric approach.
2.1 PowerTrials Application
Within Oracle Health - Cerner, the PowerTrials application integrates research processes into the Cerner electronic health record (EHR) workflow, sharing clinical research and care data with researchers and clinicians to help improve patient safety, study recruitment, and research processes. It also provides clinicians with awareness of patient research participation using flags and notifications. These alerts are sent to the study team members to help improve safety for patients who are currently enrolled in a study and who have had an encounter in the emergency department (ED) or another non-research visit that may be related to an adverse event. PowerTrials compares EHR records and study criteria to inform researchers and clinicians of potential study candidates. It uses the existing EHR order catalog and PowerPlan functionality to develop protocol-based plans of care to identify the charges as standard of care or study care that is reimbursed by the sponsor. With PowerPlan, research protocols and orders can be grouped into multiphase plans of care, enabling researchers to drive the procedures and assessments that need to occur at each visit. These features facilitate accurate research billing and comply with the Centers of Medicare and Medicaid Services (CMS). The application facilitates trial management and trial oversight activities by tracking protocols, protocol amendments, sponsor information, key contacts, and consents, and enables patient safety by providing clinical trial information to clinicians through the Clinical Research tab in PowerChart. Importantly, PowerTrials allows institutional review board (IRB) interactions to be documented and tracked as trials progress. Sites can manage numerous trials of varying size and complexity and can track accrual metrics and study milestones across all protocols or at the protocol/patient level. Standard reports are available to support operational and therapeutic oversight activities. Summary reports can provide a snapshot of research activities as well as trends across protocols. Prescreening enables clinical trial enrollment by unifying the patient care processes with the identification of clinical research candidates. Its screening engine compares inclusion and exclusion criteria to patient data in the system and supplies a prescreened patient list to clinicians and researchers of patients who meet initial eligibility requirements. Protocol-specific screening components support 2 distinct workflows for identification of potential research candidates: 1) screening protocols against a specified patient population in the research workflow and 2) screening a patient against available protocols in the clinical workflow. PowerTrials has the ability to receive protocol and patient enrollment information from external clinical trial management systems (CTMS). This capability enables institutions with existing CTMS to achieve the positive impacts and efficiencies of integrating research into the clinical workflow without entering patient-related information into two different systems.
2.2 Study Design
This cross-sectional, single-center study enrolled clinicians employed on or before September 2024. As part of the rollout short 10-15-min video overviews (high-level overview of PowerTrials, components, benefits of use, integration across the system for trials across sites, user roles, key staff, questions post roll-out) was created. The goal was participation by providers, nursing, clinical staff, pharmacy, lab, radiology, finance, and care management. Critical topics covered during initial application use included alerts and notifications, banner bar interpretation and action, the clinical research field for appropriate identification, emergency department notification and interpretation/action, and general knowledge of access and tasks. Two training modules were created to enable organizational change and integration of the PowerTrials application. For all staff, including general staff members who do not work daily with clinical trial patients, an introductory PowerTrials Clinical Research Training Module 1 was created. A second more advanced module was created to introduce the features clinical research staff would encounter with the new application in the EHR. Module 1 (introduction) included 365.25 seconds of narrated content and a demonstration with five post-training quiz questions. Module 2 (PowerTrials use) included 1,095.85 seconds of narrated content and demonstration of the application and included seven post-training quiz questions.
2.2 Experimental Set-up
Sticky by Tobii is a solution for large-scale quantitative research using eye tracking. The platform is integrated with online survey engines enabling distributed data collection set-up with a quick turnaround time. The platform gathers the data by showing media to a sample of the target audience and tracks their eye movement as they complete an assigned task. All participants were seated at computers with integrated webcams in a noise-free environment. Staff assisted participants with webcam positioning and audio volume. Walls were neutral and glasses were removed if possible. Once calibration (Figure 1 left) was completed, participants were instructed to complete the two training modules (Figure 1 right).
2.3 Statistical Analyses
Descriptive and inferential analyses were completed using Minitab (V22, Minitab LLC, State College, PA). Means, standard deviations, and medians were calculated for all continuous variables, and counts and relative frequency were calculated for all categorical variables. Normality was assessed for all variables of interest to determine if parametric or non-parametric modeling was needed. Parametric analyses were completed using mixed-model regression and analysis of variance (ANOVA) with adjustment for the baseline characteristics and other demographic variables for the primary and secondary outcome measures. Gaze and time-series analyses were used to assess trends over time in each Areas of Interest (AOIs) identified in the training (Figure 2). The level of significance was set at an alpha of 0.05.
3 Results
3.1 PowerTrials Workflow Analysis
First the team completed a workflow analysis for integration into current hospital systems. The research study process alert works in tandem with an indicator (Process Alert) on the Banner Bar of the patient chart. Patients enrolled in a study built in the PowerTrials module will display this alert. Next, the Clinical Research Tab provides a location for the research team contact information in the patient/participant chart for the care team who may have questions or concerns about the patient. The "On Research Study" hyperlink informs staff opening the chart that the patient is enrolled in research and that there is additional information they can view related to the study in the Clinical Research Tab which contains uploaded study information. PowerTrials uses the existing EHR order catalog and PowerPlan functionality to develop protocol-based plans of care to help delineate the charges between standard of care or sponsor paid. With PowerPlan, research protocols and orders can be grouped into multiphase plans of care, enabling researchers to drive the procedures and assessments that need to occur at each visit. These features facilitate accurate research billing and comply with the Centers of Medicare and Medicaid Services (CMS). The general study information is uploaded to each participant's chart enrolled in the PowerTrials application. Study Information: study contact information, a prohibited medication list, or other instructions for clinical staff regarding the patient's care are published for easy identification. Finally, PowerTrials creates discernable information related to clinical trial participation, management of patient care, timely alerts and notifications. Both study start-up (Figure 3) and the maintenance and combined future state (Figure 4) process maps were completed.
3.2 PowerTrials Training and Metrics
A total of 16 research participants completed Module 1 and 24 completed Module 2. On average, participants directed more visual attention to key text (2,707 fixations) compared to random images (390 fixations) and focused more on central screen content (1,995 fixations) than on side content (512 fixations). Negative emotions were noted more frequently (38.6%) than positive ones (0.4%) based on emotional intensity. Pop-up windows and audible explanations effectively captured participant attention. Quiz results showed higher scores for the PowerTrials interface training (91.3/100) than for the clinical trial information training (85/100). During Module 1, end users experienced engaged efficiently with basic ClinicalTrials.gov information as well critical terminology in the conduct of clinical trial research as deemed by the National Institutes of Health. Of the seven core emotions, most participants were neutral while expecting periods of sadness, puzzlement and surprise. Most of the user engagement was negative resulting in a negative valence ranging from -42 to -24.
Module 2 dove much deeper into the nuances of the changes in the EHR as well as the PowerTrials integration. End users had much more expose of EHR changes and information processing. While most end users' neutral facial expressions there were significant periods of sadness, puzzlement and surprise. Minimal disgust, fear, and joy were observed. Most displayed a negative mood from 33 to 40 and valence was negative from -44 to -29.
4 Conclusions
Healthcare is an ever-changing industry; therefore, it was no surprise to find that clinical staff are overwhelmed by the substantial changes (technological or otherwise) in complex work environments. End user interactions with the EHR chart data were substantial in Module 2, enabling better understanding of the new application integration. These methods could be leveraged to enable better uptake of the application for use in the EHR. Using participants' emotional states by interpreting facial movements captured via webcam, the team can modify the training and user interface to best fit the workflow. A facial expression being present along with the onset and offset time of this segment, its duration, information on whether the expression was classified as short or long, as well as a running number counting the instances, allows precise customization of the user interface and training for better staff uptake and competency. By identifying the seven core emotions: joy, anger, fear, surprise, sadness, contempt and disgust by capturing subtle muscle movements and micro expressions allows the team to understand bottleneck areas in the user interface, user experience shortcomings and workflow challenges. Valence and engagement are also crucial metrics for understanding emotional responses. Valence represents the overall emotional tone, ranging from negative to positive, to understand changes in the user interface and its comfortability and user acceptance over time. These methods helped the implementation team assess user experiences, product preferences, and brand perception with the overall goal to optimize engagement as a measure of the level of exprcssiveness/involvement.
Acknowledgements
This project was supported by a research contract with Genentech, Inc., a member of the Roche Group (Grant # 423-1318) and, in part, by the Billings Clinic Collaborative Science & Innovation supported and this research. This project would not have been possible without the dedication of the organization's clinical staff or its leadership. Many thanks to the Oracle Health staff for assisting in this rapid implementation project. This was a nurse-led and nurse-designed study to facilitate and support the evidence and progress of nursing science and the progress of reducing barriers to participation in clinical trials by rural and frontier residents.
References
[1] I. Gabutti, C. Colizzi, and T. Sanna, "Assessing Organizational Readiness to Change through a Framework Applied to Hospitals," Public Organization Review, vol. 23, 04/06 2022, doi: 10.1007/sl 1115-022-00628-7.
[2] C. J. Grimolizzi-Jensen, "Organizational change: Effect of motivational interviewing on readiness to change," Journal of Change Management, vol. 18, no. 1, pp. 54-69, 2018.
[3] B. J. Weiner, "A theory of organizational readiness for change," in Handbook on implementation science·. Edward Elgar Publishing, 2020, pp. 215-232.
[4] C. D. Willis et al., "Sustaining organizational culture change in health systems," Journal of health organization and management, vol. 30, no. 1, pp. 2-30, 2016.
[5] D. Mascia, F. Morandi, and A. Cicchetti, "Looking good or doing better? Patterns of decoupling in the implementation of clinical directorates," Health Care Management Review, vol. 39, no. 2, pp. 111-123, 2014.
[6] A. M. Nunes and D. C. Ferreira, "The health care reform in Portugal: Outcomes from both the New Public Management and the economic crisis," The International journal of health planning and management, vol. 34, no. l,pp. 196-215,2019.
[7] H. S. R. Sawitri and S. Wahyuni, "Readiness to Change in the Public Sector," International Journal of Business and Society, vol. 19, p. 259, 2018.
[8] V. Vaishnavi, M. Suresh, and P. Dutta, "A study on the influence of factors associated with organizational readiness for change in healthcare organizations using TISM," Benchmarking: An International Journal, vol. 26, no. 4, pp. 1290-1313, 2019.
[9] Ç. Sıcakyüz and O. H. Yüregir, "Exploring resistance factors on the usage of hospital information systems from the perspective of the Markus's Model and the Technology Acceptance Model," Journal of Entrepreneur ship, Management and Innovation, vol. 16, no. 2, pp. 93-131, 2020.
Copyright Institute of Industrial and Systems Engineers (IISE) 2025