1. Introduction
Clinical trials are essential to study the efficacy and risks associated with drugs; however, they are costly and time-consuming, occasionally requiring years for completion. A recent study found that the average cost of development of a novel drug from drug discovery to the marketing approval of a product is between $2 billion and $3 billion and can take anywhere from 12 to 18 years, with clinical trials being the most costly and time-consuming phases of the entire process [1]. The burden of such conventional processes for whole-drug development has become even more challenging with the coronavirus disease 2019 (COVID-19) pandemic, especially because of the difficulty in recruiting and retaining clinical trial participants [2,3,4]. Because of the limitations imposed by COVID-19, such as self-isolation, site closures, and travel restrictions, as of October 2021, more than 2100 clinical trials have been reported to be explicitly suspended [5,6,7].
To overcome these difficulties, attempts have been made to utilize digital technologies, including Internet of Things (IoT) and patient-generated health data (PGHD) from devices such as biowearables, smartphones, and home medical devices, and to execute decentralized clinical trials (DCTs) [8,9,10]. The proliferation of these devices is expected to accelerate patient selection and adherence to trials. In particular, attempts to increase the efficiency of clinical research and medical fields using digital health technology have expanded exponentially as of 2020 [9,10,11].
Clinical research professionals (CRPs) at trial sites are responsible for collecting patient data and evaluating drugs. Although data collection is very important in clinical trials, the collection of accurate data is difficult in practice. In particular, for outpatient clinical trials, although frequent visits to the institutions can facilitate data collection, they cause inconvenience to the participants, make it difficult to recruit patients, and increase the possibility of trial discontinuation by participants. The collected data may also be questionable. In many cases, the data report is delayed until the next visit. Therefore, the collected data are often reported to be of poor quality, missing details such as the occurrence of the event itself, the time, and the reaction of the participant [12].
At present, to solve these difficulties in data collection, various technologies for traditional clinical trials and DCTs are being implemented haphazardly. However, most of these technologies and devices have been developed from the perspective of sponsors requesting clinical trials [13,14]. Although, like many other countries, South Korea, where this study is conducted, ranks sixth (3.68%) in the industry-sponsored trials of 2020 and third (4.5%) in single-site trials. The number of trials is also increasing. However, the proportion of investigator-initiated trial among all the ongoing clinical trials is declining. The impact of investigator-initiated trials in the clinical trial field is decreasing [15].
Thus, there has been no application development reflecting the needs of CRPs conducting clinical trials at actual institutions. Therefore, CRPs have to deal with the burden of using and adapting different platforms provided by various sponsors for clinical trials and educating participants [16].
The CRPs at trial sites who monitor the participants and evaluate drugs are currently exposed to various clinical trial management systems, ranging from traditional systems to systems based on the latest technologies. However, it is difficult to find a system specialized for efficient trial and patient management that can allow CRPs at the trial site to operate in the desired way. In this scenario, the development and introduction of a new trial management system based on the needs of the CRPs at the trial site can save cost and time and yield more accurate clinical trial results quickly by using PGHD. Thus, we intended to develop a real-time clinical trial management application that reflects the needs of medical staff in clinical trials to facilitate the broader application of digital health technology to actual clinical trial sites.
2. Materials and Methods
The development of the real-time clinical trial monitoring system was guided by the waterfall model in the software life cycle (SDLC). SDLC is a methodology used to create high-quality software by utilizing clearly defined processes. As the first SDLC approach used in software development, the waterfall model shows the software development process in a linear sequential flow. This model is simple and easy to understand and use, and the entry and exit criteria are well defined. Therefore, the model can deliver software with quality based on a systematic process, especially when various experts participate in the development, as in this study [17,18]. The applied model had four stepwise phases, as shown in Figure 1, and each phase is described as follows.
CRPs from the Seoul National University Hospital Clinical Trial Center were recruited to verify their application needs. Focus group interviews (FGIs) were conducted by dividing the seven recruited volunteers into two groups: research doctors and clinical research coordinators. The ideal sample size for a focus group interview varies according to the literature [19]. In this study, the sample size was selected to ensure less than 10 people per group and more than two groups per concept, in accordance with previously described criteria [20]. The purpose of the application to be developed was explained to the interviewees, and the needs of the planned application were collected based on clinical trial situations and application functions. All seven CRPs (three doctors and four research coordinators) participated in the FGIs. Through structured open-ended questions, requirements such as essential needs and functions to be included in the application were investigated using the FGIs. A qualitative thematic analysis was used for data analysis. The data were analyzed by grouping the collected data into similar concepts and then categorizing them [21].
The design of an application system needs to be developed to fully adopt the practical needs of clinical trials; therefore, a stepwise approach to polish the system was applied. The function and structure of the application were designed considering the needs of CRPs collected through the FGIs. The main menu of the application was designed, followed by the construction of the information architecture and wireframes. The user interface was modified to reflect the opinions of the CRPs from the FGIs.
Wizard of Oz (WOZ) prototyping was performed with CRPs by using the user interface of the application to improve the user experience. Before actual development, WOZ was a way to test usability through role-playing with mockup software. Likewise, role-playing to increase usability was performed by clinical trial experts and a team of software engineers responsible for the development of application software. Based on the results of WOZ prototyping, the usefulness and efficiency of the application user interface were confirmed before actual production. Unit tests were performed to check the programming errors and usability problems of the initial version of the prototype application, and the revised version of the prototype application was made based on the unit test results [22,23].
To test all of the functions in the revised version of the prototype application, task scenarios were developed as a heuristic evaluation for a total of 48 tasks in two detailed scenarios. Step-by-step, each scenario was designed to accomplish tasks, including login, the input of adverse reactions and self-reports, and wearable device connection. Nelson’s heuristic principle is the most commonly employed principle for heuristic evaluation; however, in this study, Joyce’s SMART heuristics (short for smartphone) [24], which was designed with consideration of the mobile environment, was used instead. The severity of the problem was measured using a three-level scale, which can clearly and scientifically quantify the level of the problem and collect additional viewpoints [25,26]. Heuristic evaluation for the revised version of the prototype application was conducted not only with the participants from WOZ but also newly recruited CRPs to control for possible bias from the engagement for the application development.
The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of Seoul National University Hospital (protocol code 2011-114-1173; date: 23 September 2021). From 6 September to 17 September 2021, subjects were recruited through convenience sampling, and heuristic evaluation was carried out. Written informed consent was obtained from all subjects involved in the study.
3. Results
3.1. Phase 1. Requirements Analysis
Thematic analysis was performed using the FGI data obtained from CRPs, and the collected needs are shown in Table 1. The opinions collected were classified into seven categories. Among these categories, four categories were related to the function of the application, such as the need to record adverse reactions or concomitant drugs, to give remote feedback to the patients, and patient health record sharing; the other three categories were related to the design and composition of the application, including needs for screen menu, format standardization, and design requirements.
3.2. Phase 2. System Design
Based on the analysis of FGI results, the main menu of the application was designed to comprise health management, laboratory test results, medications, meals, concomitant medications, adverse reactions, questionnaires, and My Alarm. To optimize usability, all menu items were displayed on the main screen, which was the first screen after login [27]. The design of the information architecture of the application is schematically illustrated in Figure 2a, as applied to the menu in needs and the procedures of clinical trials.
The rows of the information architecture of the main screen are ordered according to the general procedures of clinical trials, such as recording symptoms, checking test results, entering medicine records, etc. Columns of the information architecture were organized by listing the contents to be recorded in each menu.
A wireframe with more detailed descriptions of buttons and functions was created, as shown in Figure 2b. The mobile application screen was designed to be intuitively understandable, as shown in the lower-left panel. The global navigation bar is located at the top of the screen, and management items such as weight, blood pressure, daily steps, and blood sugar level are displayed in a single row for easy recognition. Each item can also have a separate graph display with the most recent data. The value of each item can be manually entered by the user, and data is linked with the Samsung Health app, so when using a wearable device or another measuring device that works with the Samsung Health app, it can also be entered through the device. The user interface suggested for the trial version application was developed by adopting the CRPs’ feedback to remove unnecessary text and medical terms to make the screen less complicated and to include pictograms for easy understanding for non-professional trial participants.
3.3. Phase 3. Implementation: System Development
Through role-playing interactions using the WOZ prototype, the functions and configuration of the prototype application were adjusted and enhanced. Some of the initial functions were changed (e.g., the menus were rearranged to account for the clinical trial process and for the integration of duplicate menus). Data collected for concomitant medications and adverse responses met different data specifications; therefore, the two menus were designed to be separated. On the other hand, vital sign data from external devices, such as smartwatches and wearable devices, were combined with the symptom records. Since the clinical trial patient participants were not clinical professionals and were not familiar with the terms used in general clinical trials, the titles of the health report and reminder menus were changed to include more easily understandable words. The final menu lists and functions are shown in Table 2. The network comprises an external network where users’ input data, an internal network used by medical staff, and a demilitarized zone (DMZ) server for the security of the internal network, as shown in Figure 3. The application server is responsible for data processing in the DMZ, and the batch server only performs the function of detecting arrhythmia in the DMZ. The application programming interface (API) server allows communication between the internal network and DMZ; the webserver manages data processing inside the internal network, and the database stores all the data that are generated. The cloud server outside the network is responsible for relaying the user’s electrocardiogram (ECG) data captured from the wearable devices in real-time. The data are transferred in the following order: Data generated by the user → application server → API server → database storage → web server → medical check.
ECG data are transmitted through three paths. Raw ECG data are transmitted in the order mentioned above, whereas the arrhythmia detection algorithm runs in the batch server and transmits the result to the API server. Finally, for ECG streaming, the data are sent to the external cloud, and the CRP accesses the cloud from the internal network to check the streaming data.
3.4. Phase 4. Testing and Evaluation
A heuristic evaluation to identify the major usability issues and scope for improvement was conducted with the revised version of the prototype application. Five participants from the WOZ prototyping stage and six newly recruited CRPs participated in the heuristic evaluation, and the results of the evaluation are summarized in Table 3.
A scale ranging from 1 to 3 with 1 being a minor error and 5 being a critical error. There was no significant difference in the heuristic evaluation results between the existing participants and the newly recruited CRPs.
None of the heuristic evaluation items indicated critical usability errors since more than half of the evaluators perfectly used the application for each test item, which confirmed that the revised version of the prototype application could be practically applied to clinical trials. When duplicate errors were allowed for each item, the third and eighth errors were confirmed eight times each. Next, the sixth, seventh, and eleventh errors were confirmed five times each. In particular, among heuristic three errors, “errors that make it difficult to proceed with the scenario,” were identified by five evaluators and confirmed as the most frequent error. Three participants (27% of the participants) pointed out that “the input button is hidden or difficult to select,” and it was confirmed and evaluated as a “fatal error” with three points.
4. Discussion
In this study, to develop a real-time monitoring application, a stepwise approach was applied to improve usability, starting with an analysis of needs. The initial FGIs for needs analysis identified requests for “side effects and adverse reaction identification services”, “concomitant drug identification capabilities”, and “remote feedback functions”. The primary goal of a clinical trial is to assess the benefit-to-risk ratio of the drug or treatment under consideration [28]. The therapeutic benefits of an agent, which represent its impact, can be determined within a predictable range. Simultaneously, the risks and adverse effects should be investigated in consideration of the causal link between adverse events and clinical trials. In this regard, the collection of reliable data for adverse events is crucial but quite difficult [29,30]. Thus, the function request for “side effects and adverse reaction identification services” appears to be a way to solve the difficulties associated with clinical trials.
In many sponsor-initiated trials, predominantly clinical trials, the information and communication technology devices and software used for the clinical trial are usually developed by the sponsors and applied to the clinical trial sites. This system forces the trial site’s clinical trial professionals to learn to operate a new device or system and to educate the participants whenever they receive a request [31]. In contrast, the functions of an application can be learned relatively easily if the application design matches the system of the user’s institution, emphasizing the importance of a similar structure [32]. Despite the fact that personal health records in the hospital information system (HIS) are now required in clinical trial research, a number of obstacles, such as the lack of interoperability between clinical trial research systems and HIS, make their usage challenging. This may indicate the need for HIS compatibility [33,34,35]. Moreover, though the target group of each clinical trial varies depending on the stage and type of the trial, healthy volunteers participating in the phase 1 trial are relatively easy to generalize and constitute the largest number of participants across clinical trials. This indicates the need for application development based on its use in healthy adults. Especially in specialized domains, such as medical systems, the participation of actual users in program development is crucial. In this regard, visualizing the contents of the developed application, information architecture, and wireframes to encourage the involvement of CRPs who were not specialists in software development facilitated easy participation.
From the developer’s point of view, the use of the WOZ methodology for communication with the CRPs needs to be actively considered. In particular, although it was not possible to conduct research on real clinical trial participants in this study, it is a unique experience for CRPs to experience the position of participants through the WOZ process. In fact, the end general usability can be greatly increased through the WOZ process because the actual end-users are generally accepting of the program that CRPs provide, and users often do not give active and negative feedback to the CRPs in the clinical environment.
For interoperability and security, the creation of applications with Fast Healthcare Interoperability Resources (FHIR) as a standard was considered [36,37]. However, because of security reasons as well as practical difficulties in recruiting a technician with FHIR-based production experience, interoperability with the hospital network was not implemented.
Various solutions are also required for processing ECG data. For real-time ECG streaming, the limited server resources and bandwidth with the DMZ server caused delays in streaming. This problem was solved by transmission using a cloud server. However, this approach introduced a security issue because the cloud server was not located in the internal network. To solve the security issue, the Amazon Web Services Cloud, which is known for its relatively stable security among cloud services, was used. Only the function of viewing the ECG graph transmitted by the cloud was performed in the internal network, effectively blocking other data connections between networks. The arrhythmia detection algorithm also encountered resource issues. This problem was solved by physically separating one DMZ server into two logical servers, configuring the batch server, and processing the algorithm in the batch server. Thus, problems that occurred during the actual development process were solved within the limited available resources.
Heuristic evaluation was used to confirm the direction of improvement. In the heuristic evaluation, various categories of tests were planned, although the actual results indicated the importance of improving the overall usability of the application on the basis of individual errors rather than classifying the errors by category. For example, if an error occurred because the button on the screen was hidden, some participants reported it as a design error, whereas others considered it a configuration error. In this regard, developers should be careful about developing applications on a subjective basis without fully reflecting the needs of users. The most common complaint identified in the heuristic evaluation was that the input window and text were too small. Thus, the evaluation suggested that animations or design elements used to improve aesthetics may not benefit end-users who frequently use the application. However, the positive responses to the screen composition and other aesthetic aspects indicated the importance of identifying a compromise for these aspects. It was considered that the CRP group newly participating in application development would evaluate the application from a different angle than the CRP group that continued to participate in application development, but there was no significant difference in the heuristic results between the two groups [38]. This is because the CRP group relatively familiar with applications is not an expert group in application development. It is thought that these results came from the fact that both groups have in common that they are CPR.
The developed application is designed to input data related to the site’s frequently performed clinical trials according to the needs of CRPs, and data related to specific COVID-19 symptoms were added to accommodate the needs created by the pandemic. To improve the suitability of the application for various clinical trial situations, the addition of input fields and linkage with hospital data, such as laboratory and imaging test results, will be necessary. In addition, although the unit test and heuristic evaluation confirmed that the data entered by the evaluators according to the manual were stored without errors in the server, the simultaneous transmission of heavy and bandwidth-consuming data, such as ECG streaming data, has not yet been tested. Further studies are needed to confirm the fidelity of data transmission.
With the increasing importance of remote clinical trials, a clinical trial application that completely replaces the need for the direct participation of CRPs in clinical research is expected to control the cost escalations and unnecessary period extensions caused by traditional clinical trial conduct.
Despite the various implications of this study, it has several limitations. Technically, a direct link between FHIR and hospital HIS was not implemented, and only the Android version was produced, so it cannot be used in the iPhone environment. The limitations of this study are as follows. First, its use has not been verified in actual clinical practice using the developed application. Second, no direct usability evaluation was performed on clinical trial subjects during the manufacturing process. Third and last, it has not been tested for direct differences compared to other sponsor-led applications.
5. Conclusions
In this study, we developed an application to address the difficulties associated with subject management in traditional clinical trials. After the development of the application, a heuristic evaluation was performed to reflect the user demands of existing clinical trial workers. These evaluations made it possible to confirm various consistencies in the application functions and user interface. Unlike other studies, this study explains the researcher-led application development process in great detail and provides insights that were gained from each development process. In addition, the fabrication process described in this study will serve as a basis for the development of similar applications. In the future, additional real user testing and data safety studies will be needed, and the efficiency of clinical trial management is expected to improve if the application streamlined through such evaluations is applied to actual clinical trials.
Conceptualization, H.R.; resources, H.R.; methodology, H.R. and M.P.; validation, H.K.; data curation, M.P. and H.R.; writing—original draft preparation, H.R.; writing—review and editing, H.R. and M.P.; visualization, H.R. and W.Y.; supervision, K.H.K.; project administration, K.H.K.; funding acquisition, K.H.K. All authors have read and agreed to the published version of the manuscript.
This research was funded by the Korea Health Technology R&D Project through the Korea Health Industry Development Institute funded by the Ministry of Health & Welfare, Korea, grant number HI19C0790.
The study was conducted in accordance with the Declaration of Helsinki and approved by the Institutional Review Board of Seoul National University Hospital (protocol code 2011-114-1173; date: 23 September 2021).
Written informed consent was obtained from all subjects involved in the study.
The data that support the findings of this study are not publicly available due to privacy and ethical concerns but can be provided upon request to the corresponding author.
The authors declare no conflict of interest. The funder had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Enlarge this image.Figure 2. (a) Information architecture; (b) Wireframe. ECG, electrocardiogram. BMI, body mass index.
Enlarge this image.Figure 3. Network server configuration diagram. API (Application Programming Interface) is a software intermediary that allows two applications to communicate with each other. DMZ (Demilitarized zone) is a small, isolated network located between an external and internal network for data security.
Collected needs for the real-time clinical trial monitoring system.
| No. | Needs | Details |
|---|---|---|
| 1 | Checking for side effects and adverse reactions | A function for recording side effects and adverse reactions is required. |
| 2 | Concomitant drug check | Taking photos and uploading concomitant drug function information is required to check drug relationships. |
| 3 | Remote feedback function | In addition to the traditional method of calling or texting the Clinical Research Coordinator, a function to give feedback to the patient based on the data recorded in the application is required (e.g., by analyzing a chat message). |
| 4 | Data sharing with the hospital system | A function to share data such as laboratory test results, doctors’ feedback, and a brief history of the patient, from the hospital system, is required. |
| 5 | Application menu | Separate menus to check medication, diet, concomitant medications, and adverse reactions are required |
| 6 | Standard form use | The form of the application should be based on the standard form currently used in the clinical trial center. |
| 7 | Design requirements | The design should be based on the target audience of users under 60 years of age. |
Confirmed main menu structure and functions.
| No. | Menu | Function Description |
|---|---|---|
| 1 | Self-report | Patient-generated health data (PGHD), including the user’s weight, fasting blood sugar level, blood pressure, heart rate, body temperature, and oxygen saturation, were entered and checked. |
| 2 | Medication |
Medication: A medication log, which included the name and time of each medication or treatment, was maintained. The relevant data were added to the adverse reaction menu when the participants showed adverse reactions. |
| 3 | Concomitant drug | In participants consuming over-the-counter drugs or health supplements other than the test drug, information about the time and amount of the drugs was entered. |
| 4 | Adverse reactions | When an adverse reaction occurred, the type, location, period, action method, picture of the symptoms, etc., were recorded, and the management of persistent adverse reactions was documented. |
| 5 | Symptom record | Symptom record: Cough, stuffy nose, sore throat, fatigue, headache, fever, loss of smell, loss of taste, etc. (corresponding to symptoms of COVID-19) were reported. |
| 6 | Daily |
The user’s medication, nutritional, and health measurement record items that must be entered each day are presented. |
Results of the heuristic evaluation of the clinical trial monitoring application.
| Heuristic Evaluation Contents | N (%) | Mean Score | Heuristic |
|---|---|---|---|
| Program errors that make it difficult to proceed with the scenario | 5 (45%) | 2.20 | SMART 3 |
| Errors related to the “symptom input” page configuration and screen information | 5 (45%) | 1.80 | SMART 8 |
| Inconvenience caused by the graphic method for inputting time | 5 (45%) | 1.40 | SMART 11 |
| Errors related to the “Health Report” page configuration and screen information | 5 (45%) | 1.20 | SMART 7 |
| Inconvenience caused by a hidden or difficult-to-operate input button | 3 (27%) | 3.00 | SMART 6 |
| Errors caused by unclear or missing pop-ups | 3 (27%) | 2.00 | SMART 3 |
| Errors caused by missing notifications for the ECG-related connection | 3 (27%) | 1.33 | SMART 1 |
| Inconsistent screen discomfort | 3 (27%) | 1.33 | SMART 2 |
| Errors related to the “Combination Drugs” page configuration and screen information | 3 (27%) | 1.33 | SMART 8 |
| Confusing screen configuration that allowed users to input the heart rate in the blood pressure input window | 3 (27%) | 1.00 | SMART 5 |
| Inconvenience for elderly individuals or people with reduced vision due to the small font size | 2 (18%) | 2.00 | SMART 10 |
| Discomfort caused by awkward or difficult-to-understand expressions | 2 (18%) | 1.50 | SMART 2 |
| Inconvenience caused by the lack of visibility of the configuration of the menu and tab at a glance | 2 (18%) | 1.50 | SMART 6 |
| Inconvenience caused by the keyboard window covering the screen when typing | 2 (18%) | 1.50 | SMART 10 |
References
1. Moore, T.J.; Zhang, H.; Anderson, G.; Alexander, G.C. Estimated Costs of Pivotal Trials for Novel Therapeutic Agents Approved by the US Food and Drug Administration, 2015–2016. JAMA Intern. Med.; 2018; 178, pp. 1451-1457. [DOI: https://dx.doi.org/10.1001/jamainternmed.2018.3931] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30264133]
2. Fischer, S.M.; Kline, D.M.; Min, S.J.; Okuyama, S.; Fink, R.M. Apoyo Con Carino: Strategies to Promote Recruiting, Enrolling, and Retaining Latinos in a Cancer Clinical Trial. J. Natl. Compr. Canc. Netw.; 2017; 15, pp. 1392-1399. [DOI: https://dx.doi.org/10.6004/jnccn.2017.7005] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29118231]
3. Fogel, D.B. Factors Associated with Clinical Trials That Fail and Opportunities for Improving the Likelihood of Success: A Review. Contemp. Clin. Trials Commun.; 2018; 11, pp. 156-164. [DOI: https://dx.doi.org/10.1016/j.conctc.2018.08.001] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30112460]
4. Soares, R.R.; Parikh, D.; Shields, C.N.; Peck, T.; Gopal, A.; Sharpe, J.; Yonekawa, Y. Geographic Access Disparities to Clinical Trials in Diabetic Eye Disease in the United States. Ophthalmol. Retina; 2021; 5, pp. 879-887. [DOI: https://dx.doi.org/10.1016/j.oret.2020.12.006]
5. Carlisle, B.G. Clinical Trials Stopped by COVID-19 [Internet]. The Grey Literature. 2020; Available online: https://covid19.bgcarlisle.com/ (accessed on 1 January 2022).
6. Asaad, M.; Habibullah, N.K.; Butler, C.E. The Impact of COVID-19 on Clinical Trials. Ann. Surg.; 2020; 272, pp. e222-e223. [DOI: https://dx.doi.org/10.1097/SLA.0000000000004113]
7. Hamzelou, J. World in Lockdown. New Sci.; 2020; 245, 7. [DOI: https://dx.doi.org/10.1016/S0262-4079(20)30611-4]
8. Apostolaros, M.; Babaian, D.; Corneli, A.; Forrest, A.; Hamre, G.; Hewett, J.; Podolsky, L.; Popat, V.; Randall, P. Legal, Regulatory, and Practical Issues to Consider When Adopting Decentralized Clinical Trials: Recommendations from the Clinical Trials Transformation Initiative. Ther. Innov. Regul. Sci.; 2020; 54, pp. 779-787. [DOI: https://dx.doi.org/10.1007/s43441-019-00006-4]
9. Hashiguchi, T.C.O. Bringing Health Care to the Patient: An Overview of the Use of Telemedicine in OECD Countries; OECD Health Working Papers, No. 116 OECD Publishing: Paris, France, 2020; [DOI: https://dx.doi.org/10.1787/8e56ede7-en]
10. Weinstein, R.S.; Lopez, A.M.; Joseph, B.A.; Erps, K.A.; Holcomb, M.; Barker, G.P.; Krupinski, E.A. Telemedicine, Telehealth, and Mobile Health Applications That Work: Opportunities and Barriers. Am. J. Med.; 2014; 127, pp. 183-187. [DOI: https://dx.doi.org/10.1016/j.amjmed.2013.09.032]
11. Won, J.H.; Lee, H. Can the COVID-19 Pandemic Disrupt the Current Drug Development Practices?. Int. J. Mol. Sci.; 2021; 22, 5457. [DOI: https://dx.doi.org/10.3390/ijms22115457]
12. Little, R.J.; D’Agostino, R.; Cohen, M.L.; Dickersin, K.; Emerson, S.S.; Farrar, J.T.; Frangakis, C.; Hogan, J.W.; Molenberghs, G.; Murphy, S.A. et al. The Prevention and Treatment of Missing Data in Clinical Trials. N. Engl. J. Med.; 2012; 367, pp. 1355-1360. [DOI: https://dx.doi.org/10.1056/NEJMsr1203730]
13. Inan, O.T.; Tenaerts, P.; Prindiville, S.A.; Reynolds, H.R.; Dizon, D.S.; Cooper-Arnold, K.; Turakhia, M.; Pletcher, M.J.; Preston, K.L.; Krumholz, H.M. et al. Digitizing Clinical Trials. NPJ Digit. Med.; 2020; 3, 101. [DOI: https://dx.doi.org/10.1038/s41746-020-0302-y] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32821856]
14. Kario, K.; Tomitani, N.; Kanegae, H.; Yasui, N.; Nishizawa, M.; Fujiwara, T.; Shigezumi, T.; Nagai, R.; Harada, H. Development of a New ICT-Based Multisensor Blood Pressure Monitoring System for Use in Hemodynamic Biomarker-Initiated Anticipation Medicine for Cardiovascular Disease: The National IMPACT Program Project. Prog. Cardiovasc. Dis.; 2017; 60, pp. 435-449. [DOI: https://dx.doi.org/10.1016/j.pcad.2017.10.002] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29108929]
15. Korea National Enterprise for Clinical Trials [Internet]. The Grey Literature. 18 May 2021. Available online: https://www.konect.or.kr/kr/contents/datainfo_data_01_tab03/view.do (accessed on 15 March 2022).
16. Levy, H. Reducing the Data Burden for Clinical Investigators. Appl. Clin. Trials.; 2017; 26, 17.
17. Roger, S.P.; Bruce, R.M. Software Engineering: A Practitioner’s Approach; McGraw-Hill Education: New York, NY, USA, 2015.
18. Pressman, R.S. Software Engineering: A Practitioner’s Approach; Palgrave MacMillan: London, UK, 2005.
19. Carlsen, B.; Glenton, C. What about N? A Methodological Study of Sample-Size Reporting in Focus Group Studies. BMC Med. Res. Methodol.; 2011; 11, 26. [DOI: https://dx.doi.org/10.1186/1471-2288-11-26]
20. Krueger, R.A. Focus Groups: A Practical Guide for Applied Research; Sage Publications: Thousand Oaks, CA, USA, 2014.
21. Hsieh, H.F.; Shannon, S.E. Three Approaches to Qualitative Content Analysis. Qual. Health Res.; 2005; 15, pp. 1277-1288. [DOI: https://dx.doi.org/10.1177/1049732305276687]
22. Green, P.; Wei-Haas, L. The Rapid Development of User Interfaces: Experience with the Wizard of Oz Method. Proc. Hum. Factors Soc. Annu. Meet.; 1985; 29, pp. 470-474. [DOI: https://dx.doi.org/10.1177/154193128502900515]
23. Pettersson, J.S.; Wik, M. The Longevity of General Purpose Wizard-of-Oz Tools. Proceedings of the Annual Meeting of the Australian Special Interest Group for Computer Human Interaction; Parkville, Australia, 7–10 December 2015.
24. Joyce, G.; Lilley, M.; Barker, T.; Jefferies, A. Mobile Application Usability: Heuristic Evaluation and Evaluation of Heuristics. Advances in Human Factors, Software, and Systems Engineering; Springer: Cham, Switzerland, 2016; pp. 77-86. [DOI: https://dx.doi.org/10.1007/978-3-319-41935-0_8]
25. Sauro, J.; Lewis, J. Should All Scale Points Be Labelled?. 2020; Available online: https://measuringu.com/scale-points-labeled/accessed (accessed on 1 January 2022).
26. Sauro, J. Rating the Severity of Usability Problems. 2013; Available online: http://www.measuringu.com/blog/rating-severity.php (accessed on 1 January 2022).
27. Norman, D. The Design of Everyday Things: Revised and Expanded Edition; Basic Books: New York, NY, USA, 2013.
28. Umscheid, C.A.; Margolis, D.J.; Grossman, C.E. Key Concepts of Clinical Trials: A Narrative Review. Postgrad. Med.; 2011; 123, pp. 194-204. [DOI: https://dx.doi.org/10.3810/pgm.2011.09.2475]
29. Allen, E.N.; Chandler, C.I.R.; Mandimika, N.; Leisegang, C.; Barnes, K. Eliciting Adverse Effects Data from Participants in Clinical Trials. Cochrane Database Syst. Rev.; 2018; 1, MR000039. [DOI: https://dx.doi.org/10.1002/14651858.MR000039.pub2]
30. Berlin, J.A.; Glasser, S.C.; Ellenberg, S.S. Adverse Event Detection in Drug Development: Recommendations and Obligations Beyond phase 3. Am. J. Public Health; 2008; 98, pp. 1366-1371. [DOI: https://dx.doi.org/10.2105/AJPH.2007.124537]
31. Lawry, S.; Popovic, V.; Blackler, A.; Thompson, H. Age, Familiarity, and Intuitive Use: An Empirical Investigation. Appl. Ergon.; 2019; 74, pp. 74-84. [DOI: https://dx.doi.org/10.1016/j.apergo.2018.08.016]
32. Khairat, S.; Burke, G.; Archambault, H.; Schwartz, T.; Larson, J.; Ratwani, R.M. Perceived Burden of EHRs on Physicians at Different Stages of Their Career. Appl. Clin. Inform.; 2018; 9, pp. 336-347. [DOI: https://dx.doi.org/10.1055/s-0038-1648222] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29768634]
33. Mc Cord, K.A.; Ewald, H.; Ladanie, A.; Briel, M.; Speich, B.; Bucher, H.C.; Hemkens, L.G. RCD for RCTs Initiative and the Making Randomized Trials More Affordable Group. Current Use and Costs of Electronic Health Records for Clinical Trial Research: A Descriptive Study. CMAJ Open.; 2019; 7, pp. E23-E32. [DOI: https://dx.doi.org/10.9778/cmajo.20180096] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/30718353]
34. National Academies of Sciences, Engineering, and Medicine. Reflections on Sharing Clinical Trial Data: Challenges and a Way Forward: Proceedings of the a Workshop; National Academies Press: Washington, DC, USA, 2020.
35. Parab, A.A.; Mehta, P.; Vattikola, A.; Denney, C.K.; Cherry, M.; Maniar, R.M.; Kjaer, J. Accelerating the Adoption of eSource in Clinical Research: A Transcelerate Point of View. Ther. Innov. Regul. Sci.; 2020; 54, pp. 1141-1151. [DOI: https://dx.doi.org/10.1007/s43441-020-00138-y] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/32128701]
36. Rosa, M.; Faria, C.; Barbosa, A.M.; Caravau, H.; Rosa, A.F.; Rocha, N.P. A Fast Healthcare Interoperability Resources (FHIR) Implementation Integrating Complex Security Mechanisms. Procedia Comput. Sci.; 2019; 164, pp. 524-531. [DOI: https://dx.doi.org/10.1016/j.procs.2019.12.215]
37. Saripalle, R.; Runyan, C.; Russell, M. Using HL7 FHIR to Achieve Interoperability in Patient Health Record. J. Biomed. Inform.; 2019; 94, 103188. [DOI: https://dx.doi.org/10.1016/j.jbi.2019.103188]
38. Othman, M.K.; Ong, L.W.; Aman, S. Expert vs novice collaborative heuristic evaluation (CHE) of a smartphone app for cultural heritage sites. Multimed. Tools. Appl.; 2022; 81, pp. 6923-6942. [DOI: https://dx.doi.org/10.1007/s11042-022-11991-4]
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Details
; Yang, Wooseok 3 ; Kim, Kyung Hwan 4 1 Biomedical Research Institute, Seoul National University Hospital, Seoul 03080, Korea;
2 Office of Hospital Information, Seoul National University Hospital, Seoul 03080, Korea;
3 Clinical Trials Center, Seoul National University Hospital, Seoul 03080, Korea;
4 Department of Thoracic and Cardiovascular Surgery, Seoul National University Hospital, Seoul National University College of Medicine, Seoul 03080, Korea