Machine Learning Using Institution-Specific Multi-Modal Electronic Health Records Improves Mortality Risk Prediction for Cardiac Surgery Patients

Abstract

Background: The Society of Thoracic Surgeons (STS) risk scores are widely used to assess risk of morbidity and mortality in specific cardiac surgery patients but may not perform optimally or be applicable to all individuals. Using data from a consecutive cohort of cardiac surgery patients, this study sought to develop a data-driven, institution-specific machine learning-based model inferred from routinely collected electronic health records (EHR) and compare model performance with the STS risk models.

Methods: All adult (>18 years old) patients undergoing cardiac surgery at the Mount Sinai Hospital between 2011-2016 were included. Routinely collected data, consisting of administrative, demographic, clinical, hemodynamic, laboratory, pharmacological, and procedural features, were extracted from the EHR (total # features=4016). The outcome was postoperative mortality. The database was randomly split into training and test cohorts in an 80:20 ratio for the development and evaluation, respectively. Four classification algorithms [eXtreme Gradient Boosting (XGBoost), random forest, logistic regression, and support vector machines] were compared in terms of six evaluation metrics. The performance of the final model was compared to that of the STS risk models for the seven STS index surgical procedures.

Results: In total, 6,392 patients met inclusion criteria. Overall mortality was 3.0% (n=193) in both training (n=154) and test (n=39) cohorts. The XGBoost algorithm using only variables with no missing data (# of features=336) yielded the best-performing mortality predictor. When applied to the independent test set, the predictor performed well (F-measure=0.775, Precision=0.756, Recall=0.795, Accuracy=0.986, Area Under the Receiver Operating Characteristic Curve=0.978, and Area Under the Precision-Recall Curve=0.804). Additionally, the XGBoost model consistently demonstrated improved performance compared to the STS risk models when evaluated on index-specific procedures within the test set.

Conclusions: Machine learning-based models using institution-specific multi-modal EHR data may provide improved performance in predicting mortality for individual adult cardiac surgery patients as compared to the standard-of-care STS risk scores derived from population-level data. Institution-specific models can provide insights complementary to population-derived risk prediction measures to aid in informed patient-level decision making.