Content area
Purpose
We present a machine learning-based online recommendation system for stroke risk assessments. With this tool, users will be able to take proactive steps in managing their health by predicting stroke risk based on diverse data input, providing transparent and reliable risk factor interpretations, and helping healthcare professionals make informed clinical decisions.
Methods
This study uses the publicly available Stroke Analysis dataset. To predict stroke risk, the CatBoost classifier is employed, while the XAI component incorporates SHAP explainer to provide insights into its reasoning. A Django-based web application allows users to upload risk factor data and receive personalized stroke risk predictions. Smartwatch integration allows continuous monitoring of dynamic risk factors. BioMistral 7B Large Language Models (LLM) is employed to create an intuitive AI medical assistant.
Results
The developed automated online recommender system is highly accurate and robust for stroke risk assessment. The CatBoost classifier shows an average AUC of 0.98. In addition to the SHAP explainer, the recommender system also integrates Google Maps, Alert System, and Q/A chatbot based on LLMs.
Conclusion
According to the study, AI-driven systems can assist in stroke risk assessment and preventive care strategies. Developing a user-friendly online recommender system provides proof of principle for an efficient and user-friendly health management tool using machine learning, explainable AI, and LLM.
Introduction
Acute ischemic stroke is a critical neurovascular condition that disrupts blood flow to the brain, causing oxygen deprivation. It is the leading cause of neurological disability and death. According to the Centers for Disease Control (CDC), ischemic strokes make up approximately 87% of all stroke cases, affecting over 795,000 people annually in North America [1]. Moreover, individuals who have experienced a transient ischemic attack (TIA) face a higher risk of recurrent strokes [2]. Early stroke risk prediction could help reduce both mortality and morbidity.
A major issue in the Canadian healthcare system is long wait times for medical appointments [3]. Research shows that compared to other Commonwealth countries, Canadians experience the longest wait times to see specialists [3]. Powers et al. [4] emphasized the urgent need for rapid pre-hospital care and assessment for patients at risk of acute ischemic stroke leading to the need for a more efficient stroke care system. Another study highlights the global need for improved stroke response systems, especially in low- and middle-income countries [5]. There may be a need to support the healthcare system with an accessible framework that provides tools for quick stroke risk predictions and continuous patient monitoring.
Previous studies have used machine learning (ML) techniques to predict stroke risk based on clinically identified factors, but these models are often considered ”black boxes” meaning that while they can be highly accurate, they provide little to no insight into their prediction-making process. This lack of transparency poses a significant challenge, particularly in healthcare, where understanding the rationale behind model decisions is critical. Clinicians are often hesitant to trust or act on recommendations from models that do not offer clear justifications, limiting their practical use. To address this, ElShawi et al. [6] proposed using explainable ML techniques to provide clinicians with insights into the model’s predictions. Kokkotis et al. [7] applied SHAP to predict ischemic stroke risk using imbalanced data and explained the outcomes of the ML model by highlighting key risk factors.
Wearable devices have also been previously explored for stroke risk assessment [8]. Chen and Sawan conducted a comprehensive review on the integration of multimodal wearable devices in stroke risk prediction, stressing its importance in early detection and prevention strategies [8]. Soon Hyeong et al. [9] made a significant contribution by introducing an intelligent health-monitoring system capable of detecting abnormal movements using accelerometer sensor data.
Large Language Models (LLMs) with natural language processing (NLP) abilities can interpret human-made texts (prompts), making interactions easier for everyday users. Zhou et al. [10] emphasized the importance of applying NLP in the medical field to create systems that can substantially aid healthcare. When fine-tuned appropriately, such models can narrow the gap between AI and human-like understanding [11].
However, LLMs are prone to hallucinations [12]. To improve accuracy and reduce hallucinations, fine-tuning is required [13]. While BioGPT [14], a recent iteration of the GPT architecture fine-tuned for medical applications, handles text completion tasks well, it lacks the conversational capabilities of more advanced models like GPT-4 [15] or Llama 2 [16]. Furthermore, proprietary models like GPT-4 raise concerns about data privacy risks [17], a key issue in medical applications. Also, GPT-4 has yet to undergo fine-tuning for biomedical purposes, limiting its effectiveness in specialized medical domains.
In a previous study, Bolatbekov et al. [18] employed the CatBoost ML algorithm [19] and the Shapley Additive exPlanations (SHAP) tool to design an explainable machine learning model for early heart attack risk prediction. The system was built around a web application based on the Django framework [20], enabling both clinicians and regular users to input data manually or automatically transmit dynamic data (heart bit, blood pressure etc.) via a smartwatch. The system then generated results that assist in heart attack risk stratification. However, it has not been applied to stroke risk prediction yet.
The primary goal of this project is to develop an accurate and interactive tool for both everyday users and healthcare professionals for stroke risk stratification. This paper outlines the methodologies used in creating a stroke risk management framework, along with the results and discussion. We explore the use of techniques like ML, LLMs, and software development tools. We believe that advancements in LLMs can bring us closer to achieving our objective. In line with a recent study by Yanis et al. [21], we integrated BioMistral 7B into our framework as it was designed specifically for the medical domain [21]. We wanted to enhance the smart recommender system to provide personalized health advice and answer stroke-related questions, so that it can be a reliable AI assistant that can ease the strain on the healthcare system while empowering individuals to proactively manage their health.
Methods
System Overview
In this study, we aimed to build an automated online recommender system for predicting stroke risk levels based on the clinical risk factors of individual patients. The system utilizes Machine Learning (ML) techniques for risk prediction. Explainable AI (XAI), specifically using the SHAP method, was employed to identify and evaluate the key risk factors that influenced the predictions. The system is accessible through a web application built on Django framework and a cloud server is used for data storage and security. Moreover, we developed methods for dynamic data acquisition using smartwatch technology. Based on user input and dynamic data acquisition from a smartwatch, the system generates patient-specific explanation reports and graphical representations of stroke risk levels and key risk factors. Alongside stroke classification and risk prediction, we have introduced features, such as an alert system, hospital finder, and a BioMistral 7B-based chatbot to answer client’s specific queries. Figure 1 illustrates the software architecture. Subsequent subsections elaborate on the system’s components and their integration.
[See PDF for image]
Fig. 1
The workings of the stroke risk assessment framework, demonstration transmission of data from smartwatch through API services and web application, encapsulating the ML model and user- framework interaction
Stroke Risk Factors and Dataset
To build the classification model, we used the publicly available Stroke Analysis dataset [22, 23] composed of 4,798 patients. The data was split into independent training (80% − 3,838 patients) and test sets (20% − 960 patients). The data consisted of stroke risk level output and 10 factors associated with stroke, such as gender, age, body mass index, smoking status, cholesterol level, systolic/diastolic blood pressure, glucose level, Thoracic Outlet Syndrome (TOS), and Modified Rankin Scale (mRS) in Table 1. The dataset consisted of a total of five static attributes (e.g., gender, age, Modified Rankin Scale (mRS)) that are non-modifiable, and six dynamic attributes (e.g., cholesterol level, systolic and diastolic blood pressure, body mass index (BMI), smoking status, and glucose level) that can vary over time. The risk class labels were described as no stroke risk, low stroke risk, moderate stroke risk, and high stroke risk respectively.
Data Preparation
The data preparation steps encompassed data cleaning, feature selection, data transformation, data balancing, and data splitting. Feature selection was guided by clinical specialist decisions, resulting in the removal of specific attributes from the original dataset such as the Health Stroke Scale (NIHSS) and paralysis status. Data cleaning involved the elimination of medically insignificant information, such as patient-assigned negative mRS values. Data transformation entailed converting the smoking attribute into a binary column, simplifying the process by representing non-smokers as 0 and smokers as 1. We assessed the impact of this balancing technique on model performance using three key metrics: ROC AUC score, Cohen’s Kappa score, and F1 score. We recognize that undersampling reduces the overall dataset size that could potentially impact model robustness. For this reason, we implemented SMOTE (Synthetic Minority Over-sampling Technique) that populates the minority group to match with the majority group. SMOTE was applied only to the training set after the dataset was split to avoid data leakage. It generated synthetic examples for the minority class, addressing gender imbalance and improving model performance on imbalanced data. These strategies reduced bias leading to a more reliable model for clinical use.
Table 1. The Spearman correlation of the dataset among 10 stroke risk factors used in the analysis
Age | Gender | MRS | Systolic | Diastolic | Glucose | Smoking | BMI | Cholesterol | TOS | |
|---|---|---|---|---|---|---|---|---|---|---|
Age | 1 | 0.04 | 0.07 | 0.38 | 0.39 | -0.01 | 0.30 | 0.02 | 0.39 | -0.02 |
Gender | 0.04 | 1 | 0.12 | 0.08 | 0.08 | 0.06 | 0.06 | 0.03 | 0.09 | 0.09 |
MRS | 0.07 | 0.12 | 1 | 0.43 | 0.32 | 0.54 | 0.12 | 0.26 | 0.26 | 0.55 |
Systolic | 0.38 | 0.08 | 0.43 | 1 | 0.56 | 0.36 | 0.28 | 0.19 | 0.48 | 0.33 |
Diastolic | 0.39 | 0.08 | 0.32 | 0.56 | 1 | 0.25 | 0.32 | 0.19 | 0.48 | 0.25 |
Glucose | -0.01 | 0.06 | 0.54 | 0.36 | 0.25 | 1 | 0.05 | 0.21 | 0.20 | 0.38 |
Smoking | 0.30 | 0.06 | 0.12 | 0.28 | 0.32 | 0.05 | 1 | 0.07 | 0.29 | 0.10 |
BMI | 0.02 | 0.03 | 0.26 | 0.19 | 0.19 | 0.21 | 0.07 | 1 | 0.12 | 0.22 |
Cholesterol | 0.39 | 0.09 | 0.26 | 0.48 | 0.48 | 0.20 | 0.29 | 0.12 | 1 | 0.15 |
TOS | -0.02 | 0.09 | 0.55 | 0.33 | 0.25 | 0.38 | 0.10 | 0.22 | 0.15 | 1 |
Stroke Classification Model
The classification model in this study was built using the CatBoost classifier [19] for the four-stroke risk level classes (no risk-0, low risk-1, moderate risk-2, and high risk for stroke-3). CatBoost [19] is a ML algorithm with an ability to use categorical features by transforming them into numbers with- out carrying out any data pre-processing. Furthermore, CatBoost employs Bayesian estimators to substantially decrease the chances of overfitting caused by generic gradient-boosting algorithms.
Explainable AI (XAI)
Our risk recommender aims to provide insights and understanding into the decision-making process of ML models, thus enhancing transparency and fostering trust in their outcomes. To be considered valuable by clinicians, the CatBoost classifier must also provide insight into the reasoning behind the prediction. To combat this issue XAI was employed in the method. For this process, SHAP and LIME were used for model interpretation [6, 18].
SHAP technique is a model explanation tool that applies principles from game theory to elucidate individual predictions. By calculating the Shapley value of each feature, it is possible to gain insights into the underlying reasoning of any black-box model, regardless of its specific architecture [6]. This approach facilitates both local and global explanations. SHAP offers three primary classes namely KernelExplainer for linear models, DeepExplainer for deep neural networks, and TreeExplainer for tree- based models. In this study, the TreeExplainer was utilized to interpret the predictions generated by the CatBoost classifier.
Online Recommender System Framework
The main unit of the stroke risk monitoring system is the web application based on the Django framework [20]. This application has an integrated ML model, and it also serves as a basis for the user to generate user-specific stroke risk prediction results. The application is built on the Model-View-Template (MVT) software design architecture wherein the “Template” refers to the front-end system that the user interacts with, the “Model” refers to the.
database component in the application, and the “View” is the controller that interlinks the various objects together and provides a channel of communication between them. The system also uses Hypertext Transfer Protocol Secure (HTTPS), which is a client-server protocol used for fetching (HTTPS requests) or sending (HTTPS responses) data to the web server.
Cloud Server Deployment
As a cloud-based server, we employed AWS for its comprehensive security features and its capability to effectively secure ML-based applications. AWS provides HTTPS support services through AWS Certificate Manager, ensuring that data communication remains secure. Automatic updates and security patches are managed seamlessly via Amazon ECS, which helps minimize vulnerability risks. Additionally, we used AWS IAM, which enables role-based access control, allowing for detailed permissions for each user and enhancing data privacy. Extensive logging and auditing capabilities are facilitated by AWS CloudTrail and Amazon CloudWatch Logs, making AWS a reliable platform for Django ML projects.
The BioMistral chatbot is subsequently deployed in a private repository via Hugging Face within Amazon SageMaker, leveraging the robust infrastructure provided by AWS. The model is served to our web application through the use of an AWS SageMaker Endpoint, ensuring efficient and secure integration with the application.
Continuous Data Acquisition Model
This platform-agnostic recommendation system integrates a Smartwatch system to collect dynamic stroke risk data. Samsung Galaxy Watch 5 enables real-time patient monitoring with FDA-approved sensors for critical data collection, including blood pressure. Data is dynamically transmitted via a JSON POST request to the web application. Data is formatted into interactive tables using Airtable API as an intermediary.
Map and an Alert System
In addition to predicting patient risk, the system enhances healthcare access and management with features such as a map that uses Google’s API for geolocation services to display nearby hospitals. Also, a real-time patient health alert system was integrated, which can notify the user in real-time if any input information is abnormal. The system ensures data integrity and privacy for reliable record keeping and analysis by using a PostgreSQL database with SHA256 encryption and providing a user-friendly interface.
Integration of Fine-Tuned LLM Model BioMistral 7B
To create a holistic stroke management application, we employed the latest open-source BioMistral7B model in the system [21]. This model, with its conversational style text generation capabilities, is fine-tuned on PubMed and other medical literature data. The use of BioMistral7B in our study offers several advantages, particularly in the medical domain. As a specialized language model, it is designed to handle complex medical terminology and clinical data with high accuracy. Its ability to process large volumes of text allows for more comprehensive analysis of patient records and clinical guidelines, which enhances decision-making in healthcare. Additionally, we leverage in-context and few-shot learning techniques to further enhance the model’s adaptability and performance in various clinical scenarios. In our study, BioMistral7B was combined with explainable AI methods, such as SHAP, to make its predictions more transparent and interpretable, which is crucial for clinical use. This combination of domain-specific knowledge and explainability not only improves the predictive performance of Strokebot but also builds trust among medical professionals, as it provides insights into key factors driving the stroke model’s decisions. We integrated the BioMistral7B to create more robust and reliable AI solutions tailored for stroke applications.
Clinician Feedback
To ensure the usability and clinical relevance of the recommender system, we sought feedback from volunteer clinician, Dr. Mira Salih. Dr. Salih actively engaged with the recommender system, testing its functionalities and assessing its performance within a clinical context. Following this hands-on evaluation, Dr. Salih provided a comprehensive review, offering insights into the system’s strengths, weaknesses, and potential areas for improvement.
Results
Data Collection and Processing
In this study, publicly available data from 4,798 patients [23] was used. The dataset encompassed 10 stroke risk factors (Table 1), including systolic and diastolic blood pressure, which has been continuously collected using the Smartwatch system. This comprehensive dataset served as the foundation for the subsequent analysis.
Machine Learning Component Performance
The machine learning component of the proposed automated online recommender system exhibited promising performance in stroke risk assessment. The classifier algorithm (trained on 80% of the data set) achieved an average area under the curve (AUC) of 0.98 on the test data set (20%) when using a one-vs-all approach (Fig. 2). This high AUC score indicates the system’s ability to accurately discriminate between different risk categories. Additionally, Cohen’s Kappa score of 0.74 and the weighted average F1 score of 0.91 show a robust balance between precision and recall.
Global-based Explanation
The risk classification was further analyzed using the SHAP method. We determined the global importance of each risk factor by assigning SHAP values to every factor in the dataset. As a result, the top 10 factors that affected the final prediction were identified (Fig. 3).
Patient-Specific Explanation
In Fig. 4, the risk factors that contributed to an individual patient’s prediction are shown. The following characteristics of a 21-year- old male were used as an example: mRS = 2, systolic blood pressure = 88, diastolic blood pressure = 88, glucose = 94, smoking (0), BMI = 22, cholesterol = 23, TOS = 1. As predicted by the model, the individual has a stroke risk and the outcome matches the actual class in the dataset.
In this study, we also conducted a model consistency analysis between two model explainers, LIME and SHAP. Specifically, we focused on the identification of the top five risk factors generated by each explainer. Our findings revealed a high similarity between LIME and SHAP explanations for all risk levels with a minimum of two common risk factors identified by both explainers.
Furthermore, we quantified the degree of similarity for each risk class, yielding percentages of 88%, 97%, 92%, and 93% for four classes (0–3), respectively. For each risk level, LIME and SHAP identified risk factors with a high similarity percentage.
[See PDF for image]
Fig. 2
Multiclass ROC curve generated based on one-versus-rest using the classification model
Such consistency in the explanations indicates the accuracy of SHAP, proving the validity of its interpretation and supporting its application in risk prediction tasks.
User Interface and Dashboard
The user interface of the recommender system provided an intuitive and interactive experience for users. Through the dashboard, users could easily access their personalized stroke risk predictions and rank the identified risk factors based on importance. According to the user’s input information, the CatBoost classifier could determine the risk of stroke of the individual. Moreover, abnormal risk factor values would activate the alert system as intended. The Google Maps feature was able to locate nearby hospitals and the integrated Q&A feature based on BioMistral 7B answered basic stroke-related questions.
Clinical Reports and Data Accessibility
To enhance communication between patients and healthcare professionals, the system generates patient-specific clinical reports in PDF format. These reports contain detailed explanations of the prediction results, which may support informed discussions with medical practitioners. User registration and the implementation of the PBKDF2 encryption algorithm safeguard the confidentiality and integrity of the stored information, granting authorized clinicians secure access for diagnosis and treatment planning.
BioMistral 7B
To make our system competitive in the current AI technology race, we integrated BioMistral 7B (a fine-tuned LLM on biomedical data) that uses the patient-specific data from the classification part of the system. The created chatbot can answer stroke-related questions and tie the answers to the parameter values and stroke risk of each patient. Moreover, when given questions out of context, the chatbot can derive information from the Internet, and finish its answer with a patient’s health- related recommendation.
[See PDF for image]
Fig. 3
SHAP-based global risk factor ranking
[See PDF for image]
Fig. 4
Modified patient-specific clinical report, generated by the automated online recommender system for John Doe (an anonymous user)
Clinician Feedback Summary
”The stroke risk assessment system received positive feedback for its user-friendly design, which allowed for smooth navigation and quick access to comprehensive results. Clinically, the system demonstrated high accuracy in identifying key risk factors, especially for patients falling within the moderate to high-risk spectrum, enabling timely interventions to mitigate stroke risk. However, a notable limitation was its tendency to overestimate risk among low-risk patients, potentially leading to unnecessary anxiety. This is where the new chatbot interface can provide significant value, offering a more nuanced review of the risk factors and personalized recommendations for lifestyle adjustments or medical advice to alleviate patient concerns.”
Discussion
The online automatic recommender system utilizes ML methods to stratify stroke risk. It employs XAI strategies to make complex stroke information easily understandable for ordinary users and clinicians. The system predicts stroke risk along with associated risk factors. The system is designed with a simple web interface that is practical and publicly available as proof of principle.
Specifically, to achieve accurate stroke risk prediction, the system employs the CatBoost classifier. The XAI component of the system utilizes SHAP to explain and interpret the results. This is done by ranking risk factors based on their contribution to stroke risk. The web application is developed using the Django framework in the MVT software architecture. Users can upload their risk factor data and receive results, which can also be downloaded as a clinical report. Additionally, the system incorporates automatic data acquisition from a Smartwatch.
The stroke risk classification model in the web application achieved an AUC of 0.98, using the one-vs-all approach on the test set, which would indicate the high accuracy of the prediction. We used the SHAP algorithm to rank the risk factors and visually represent the impact of each feature on prediction. The decision was made to incorporate SHAP instead of LIME since SHAP analyses were found to be more consistent than LIME. The system could aid clinicians in monitoring key risk factors and facilitating stroke prevention. The recommender system is publicly available at the mamatjanlab website (https://www.mamatjanlab.com/stroke/).
The discussion around LLMs in healthcare applications should consider the significant advancements they represent as well as the notable challenge of hallucinations, particularly within the biomedical domain. In the case of the Stroke Predictor system, we utilized the BioMistral 7B LLM, which demonstrated superior performance across several medical evaluation datasets, including Medical Genetics, CollegeMedicine, MedQA, and PubMedQA, outperforming other open-source models [24, 25]. To enhance efficiency, Langchain was integrated with our BioMistral 7B model, dynamically injecting patient-specific data and the risk assessment generated by the CatBoost model into the conversational AI model. This process involved adjusting model parameters, such as setting the temperature to 0.3 to decrease response randomness and assigning our model the role of ”Stroke Assistant” to ensure relevant and specific responses for each patient’s context.
Advanced prompt engineering using Langchain, including the integration of few- shot learning and in-context learning, was implemented to ensure model responses are medically accurate and patient-specific. This allows the model to generalize from limited examples and adapt dynamically to different scenarios. Additionally, a memory buffer mechanism was incorporated to enhance the LLM’s context awareness.
throughout the conversation, managing conversation history to allow the AI to reference previous dialogue and maintain continuity. By doing so, the system becomes more adept at providing contextually relevant responses and reduces the likelihood of generating inaccurate or irrelevant information. The deployment of BioMistral on the Hugging Face dedicated inference service, running on AWS servers, provides a robust and scalable infrastructure for this system.
This proof of principle highlights how AI-driven systems can optimize preventive care strategies, empower patients to manage their health proactively, and support clinicians in making informed decisions. The web application allows immediate use of the system while ensuring secure patient records storage. It is imperative to note that the system is not intended for stroke diagnosis but rather serves as a pre-assessment tool to guide patients in taking preventative measures. The predictions provided by the system can offer clinicians a second point of view when diagnosing patients. Furthermore, we have enhanced the system by incorporating features such as a Find-a-Hospital tool, an alert mechanism, and a Q&A chatbot that integrates the classification model with explainable AI to provide reasoning behind predictions. This integration reduces the chatbot’s susceptibility to hallucinations.
Overall, a successful deployment of the proposed automated online recommender system highlights the potential of AI technologies in healthcare. The high accuracy scores achieved by the ML component demonstrate the system’s efficacy in predicting stroke risk. This study provides evidence that artificial intelligence integration in stroke risk assessment can enhance personalized healthcare decision-making. The developed system showcases the potential of AI-driven recommender systems to optimize preventive care strategies, empower individuals to take proactive steps in managing their health and support healthcare professionals in making informed clinical decisions.
Future Works
One limitation is the potential inaccuracies associated with blood pressure measurements obtained through wearable technology. Future improvements should focus on integrating advanced, validated blood pressure monitoring devices for a more reliable risk assessment tool. Moreover, we can improve the system by implementing specific thresholds for variables like glucose levels to enhance accuracy. For instance, excluding values below 125 from consideration as significant stroke risk factors aligns with clinical standards and refine predictive capabilities.
Acknowledgements
We acknowledge the assistance of Dr. Dilbar Salman for her help with reviewing the paper, and Ehsan Mamatjan for providing technical support in AI model integration and web design for our Strokebot.
Author Contributions
MA and SK were responsible for content organizing, and manuscript writing. SK, NA and YM were responsible for software development and performing analysis. MS tested the chatbot and made clinical suggestions. YM designed and guided the project and provided the funding. All Authors reviewed and revised the manuscript.
Funding
This study is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant (RGPIN-2023-05341).
Data Availability
We used the Stroke Analysis Data set (Bandi, et al., 2020) from Mendeley and accessed it using the link: https://data.mendeley.com/datasets/jpb5tds9f6.
Declarations
Ethical Approval
Not applicable.
Consent to Participate
Not applicable.
Consent to Publish
All authors give permission to publish the research findings.
Competing Interests
The authors declare that they have no conflict of interest.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
1. U.S. Centers for Disease Control and Prevention (2022, September 1). Stroke Facts. Retrieved March 7, 2024, from https://www.cdc.gov/stroke/data-research/facts-stats/index.html
2. Khare, S. Risk factors of transient ischemic attack: An overview. Journal of Mid-life Health; 2016; 7,
3. Canadian Institute for Health Information (2021, February). How Canada Compares: Results from the Commonwealth Fund’s 2020 International Health Policy Survey of the General Population in 11 Countries (Publication No. 119). Retrieved March 7 2024, from https://www.cihi.ca/sites/default/files/document/how-canada-compares-cmwf-survey-2020-chartbook-en.pdf
4. Powers, W; Rabinstein, A; Ackerson, T; Adeoye, O; Bambakidis, N; Becker, K; Biller, J; Brown, M; Demaerschalk, B; Hoh, B; Jauch, E; Kidwell, C; Leslie- Mazwi, T; Ovbiagele, B; Scott, P; Sheth, K; Southerland, A; Summers, D; Tirschwell, D. 2018 guidelines for the early management of patients with Acute ischemic stroke: A Guideline for Healthcare professionals from the American Heart Association/American Stroke Association. Stroke; 2018; 49, pp. e46-e99. [DOI: https://dx.doi.org/10.1161/STR.0000000000000158] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/29367334]
5. Johnson, W; Onuma, O; Owolabi, M; Sachdev, S. Stroke: A global response is needed. Bulletin of the World Health Organization; 2016; 94, pp. 634-634. [DOI: https://dx.doi.org/10.2471/BLT.16.181636] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/27708464][PubMedCentral: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5034645]
6. El Shawi, R; Sherif, Y; Al-Mallah, M; Sherif, S. Interpretability in healthcare: A comparative study of local machine learning interpretability techniques. Computational Intelligence; 2020; 37,
7. Kokkotis, C., Giarmatzis, G., Giannakou, E., Moustakidis, S., Tsatalas, T., Tsiptsios, D., Vadikolias, K., & Aggelousis, N. (2022). An Explainable Machine Learning Pipeline for Stroke Prediction on Imbalanced Data. textitDiagnostics (Basel Switzerland), 12(10), 2392. https://doi.org/10.3390/diagnostics12102392.
8. Chen, YH; Sawan, M. Trends and challenges of Wearable Multimodal technologies for Stroke Risk Prediction. Sensors (Basel Switzerland); 2021; 21,
9. Jeong, S., Shen, J. H., & Ahn, B. (2021). A study on smart healthcare monitoring using IoT based on blockchain. Wireless Communications and Mobile Computing, 2021, 9932091, 1–9. https://doi.org/10.1155/2021/9932091
10. Zhou, B; Yang, G; Shi, Z; Ma, S. Natural Language Processing for Smart Healthcare. IEEE Reviews in Biomedical Engineering; 2024; 17, pp. 4-18. [DOI: https://dx.doi.org/10.1109/RBME.2022.3210270] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/36170385]
11. Zhao, W., Zhou, K., Junyi, L., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., Du, Y., Yang, C., Chen, Y., Chen, Z., Jiang, J., Ren, R., Li, Y., Tang, X., Liu, Z., & Wen, J. (2023). A Survey of Large Language Models. ArXiv, Cornell University. https://doi.org/10.48550/arXiv.2303.18223
12. Huang, L; Yu, W; Ma, W; Zhong, W; Feng, Z; Wang, H; Chen, Q; Peng, W; Feng, X; Qin, B; Liu, T. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. textitArXiv; 2023; [DOI: https://dx.doi.org/10.1145/3703155]
13. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). Training language models to follow instructions with human feedback. textitArXiv. https://doi.org/10.48550/arXiv.2203.02155
14. Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., & Liu, T. Y. (2022). BioGPT: Generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6). https://doi.org/10.1093/bib/bbac409. Article bbac409.
15. OpenAI (2024). ChatGPT-4 (Version 4) [Large language model]. OpenAI. https://openai.com/gpt-4
16. Meta (2024). (n.d.). Llama2 (Version 2) [Large language model]. Meta. Retrieved February 11, from https://llama.meta.com/llama2
17. Nori, H; King, N; McKinney, S; Carignan, D; Horvitz, E. Capabilities of GPT-4 on medical challenge problems. ArXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2303.13375]
18. Bolatbekov, A., Salman, D., Khan, S., Mamatjan, E., Orhun, M., & Mamatjan, Y. (2024). A Smart Recommender System to Stratify Heart Attack Risk. CMBES Proceedings, 46. Retrieved July 2024, from https://proceedings.cmbes.ca/index.php/proceedings/article/view/1192
19. Dorogush, AV; Ershov, V; Gulin, A. CatBoost: Gradient boosting with categorical features support. ArXiv Cornell University; 2018; [DOI: https://dx.doi.org/10.48550/arXiv.1810.11363]
20. Django Software Foundation (n.d.). Django. Retrieved from https://www.djangoproject.com/
21. Labrak, Y., Bazoge, A., Morin, E., GourraudP.-A., Rouvier, M., & Dufour, R. (2024). BioMistral: A collection of open-source pretrained large language models for medical domains. Findings of the Association for Computational Linguistics: ACL 2024, 5848–5864. https://doi.org/10.18653/v1/2024.findings-acl.348
22. Lundberg, S; Lee, SI. A unified approach to interpreting model predictions. ArXiv; 2017; [DOI: https://dx.doi.org/10.48550/arXiv.1705.07874]
23. Bandi, V., Midhunchakkaravarthy, D., & Bhattacharyya, D. (2020). Stroke Analysis (Version 1) [Data set]. Mendeley Data. https://doi.org/10.17632/jpb5tds9f6.1
24. Tran, H; Yang, Z; Yao, Z; Yu, H. BioInstruct: Instruction tuning of large language models for biomedical natural language processing. arXiv; 2023; [DOI: https://dx.doi.org/10.48550/arXiv.2310.19975]
25. Wu, C., Zhang, X., Zhang, Y., Wang, Y., & Xie, W. (2023). PMC-LLaMA Further finetuning LLaMA on medical papers. ArXiv. https://doi.org/10.48550/arXiv.2304.14454
© Taiwanese Society of Biomedical Engineering 2024.