Critical appraisal and assessment of bias among studies evaluating risk prediction models for in-hospital and 30-day mortality after percutaneous coronary intervention: a systematic review

Abstract

Objective

We systematically assessed prediction models for the risk of in-hospital and 30-day mortality in post-percutaneous coronary intervention (PCI) patients.

Design

Systematic review and narrative synthesis.

Data sources

Searched PubMed, Web of Science, Embase, Cochrane Library, CINAHL, CNKI, Wanfang Database, VIP Database and SinoMed for literature up to 31 August 2023.

Eligibility criteria

The included literature consists of studies in Chinese or English involving PCI patients aged ≥18 years. These studies aim to develop risk prediction models and include designs such as cohort studies, case–control studies, cross-sectional studies or randomised controlled trials. Each prediction model must contain at least two predictors. Exclusion criteria encompass models that include outcomes other than death post-PCI, literature lacking essential details on study design, model construction and statistical analysis, models based on virtual datasets, and publications such as conference abstracts, grey literature, informal publications, duplicate publications, dissertations, reviews or case reports. We also exclude studies focusing on the localisation applicability of the model or comparative effectiveness.

Data extraction and synthesis

Two independent teams of researchers developed standardised data extraction forms based on CHecklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies to extract and cross-verify data. They used Prediction model Risk Of Bias Assessment Tool (PROBAST) to assess the risk of bias and applicability of the model development or validation studies included in this review.

Results

This review included 28 studies with 38 prediction models, showing area under the curve values ranging from 0.81 to 0.987. One study had an unclear risk of bias, while 27 studies had a high risk of bias, primarily in the area of statistical analysis. The models constructed in 25 studies lacked clinical applicability, with 21 of these studies including intraoperative or postoperative predictors.

Conclusion

The development of in-hospital and 30-day mortality prediction models for post-PCI patients is in its early stages. Emphasising clinical applicability and predictive stability is vital. Future research should follow PROBAST’s low risk-of-bias guidelines, prioritising external validation for existing models to ensure reliable and widely applicable clinical predictions.

PROSPERO registration number

CRD42023477272.

YS and CZ are joint first authors.

STRENGTHS AND LIMITATIONS OF THIS STUDY

• Systematic searches were conducted for recently published models predicting short-term mortality following percutaneous coronary intervention, with comprehensive introductions provided on the modelling methods, model performance, predictive factors, risks of bias and clinical applicability scores.

• Due to differences in sample characteristics, variable selection and construction of algorithms among different studies, the review only conducted a qualitative analysis and did not perform meta-analysis.

• The machine learning algorithm models included in this review involve complex algorithms and data processing procedures, which may result in risk of bias that cannot be fully assessed by Prediction model Risk Of Bias Assessment Tool.

Introduction

In recent years, influenced by factors such as an ageing population and unhealthy lifestyle habits, there has been a continuous rise in the incidence of myocardial infarction among residents.¹ This trend poses a severe threat to public health stability and further increases the burden on medical resources. Percutaneous coronary intervention (PCI), as a widely adopted method of coronary revascularisation, can quickly reperfuse ischaemic myocardium, thereby minimising myocardial damage.² Currently, with the maturation of cardiovascular interventional technologies, the number of PCI procedures has significantly increased. However, this trend is accompanied by a persistently high risk of postoperative mortality.^{3 4} Studies indicate that post-PCI mortality is influenced by various factors, including specific patient characteristics and postoperative complications, rather than being solely caused by the PCI procedure itself.⁵ Therefore, early assessment of risk factors and taking measures to avoid these risks are crucial for improving the quality of diagnosis and treatment of myocardial infarction.

Clinical risk prediction models, using medical data and statistical methods, assess the likelihood of a patient experiencing a certain health event in the future.⁶ Currently, numerous risk prediction models have been developed to address the issue of short-term mortality after PCI,^7–9 aiming to identify high-risk patients early on and optimise reperfusion strategies promptly. However, due to the current lack of comprehensive evaluations regarding detailed performance, risk of bias and clinical applicability of relevant prediction models, the clinical application of many models is subject to limitations. This review aims to systematically evaluate risk prediction models for in-hospital and 30-day mortality following PCI, intending to provide reference for the construction and application of future relevant models.

Methods

This review follows relevant guidelines and standards for the systematic evaluation of prediction models, including Preferred Reporting Items for Systematic Reviews and Meta-Analyses and Checklist for critical Appraisal and data extraction for systematic Reviews of prediction Modelling Studies (CHARMS). The study protocol was registered with the International Prospective Register of Systematic Reviews under the registration number CRD42023477272. All research personnel involved in this review have undergone uniform training in the systematic evaluation of prediction models to ensure consistency in evaluation methods.

Search strategy

A systematic search was conducted in PubMed, Web of Science, Embase, Cochrane Library, CINAHL, CNKI, Wanfang Database, VIP Database and SinoMed to identify studies on the construction of risk prediction models for in-hospital and 30-day mortality among patients after PCI. The search covered studies published from the inception of each database to 31 August 2023. Search terms were a combination of controlled vocabulary (Medical Subject Heading (MeSH) terms) and free-text terms, including “percutaneous coronary intervention[MeSH terms]“, “PCI”, “death[MeSH terms]”, “mortality[MeSH terms]”, “nomograms[MeSH terms]”, “risk assessment[MeSH terms]”, “risk score”, “risk prediction”, “risk model”, “risk tool”, “predictive tool”, “predict model” and “scoring system”. Additionally, we manually searched the reference lists of included studies to identify any potential relevant studies. The detailed search strategy is outlined in online supplemental material 1.

Inclusion and exclusion criteria

We included literature that meets the following criteria: (1) literature in Chinese or English; (2) the study subjects are PCI patients aged ≥18 years; (3) studies on the construction of risk prediction models; (4) cohort studies, case–control studies, cross-sectional studies or randomised controlled trials; (5) the prediction model contains at least two predictors. We excluded literature meeting the following criteria: (1) the prediction model includes other post-PCI outcome indicators besides death; (2) the literature lacks essential details on study design, model construction and statistical analysis; (3) the prediction model is constructed based on a virtual dataset; (4) conference abstracts, grey literature, informal publications, duplicate publications, dissertations, reviews or case reports; (5) studies investigating the localisation applicability of the prediction model or comparative research on predictive efficacy.

Literature selection and data extraction

The retrieved literature was imported into the literature management software (Zotero), where duplicate publications were initially removed by the software. Subsequently, two independent researchers (CZ and WQ) screened the relevant literature based on inclusion and exclusion criteria by reading titles, abstracts and other relevant content. They cross-checked their selections, and in case of disagreement, a third researcher (SC) arbitrated the decision regarding the inclusion or exclusion of the literature. Additionally, we developed standardised data extraction forms based on CHARMS. Two independent researchers (DX and CW) were responsible for extracting and cross-verifying the data, with any discrepancies resolved through consultation with a third reviewer (XC). The extracted data include: first author, publication year, country, study type, study population, outcome prediction, sample size, candidate variables, missing data, construction methods, model validation, model performance and final model presentation format.

Risk of bias assessment

Two independent researchers (YS and CZ) used the multifactor predictive construction risk assessment tool (Prediction model Risk Of Bias Assessment Tool (PROBAST)) to conduct risk of bias assessment and applicability evaluation for model development or validation studies included in this review. In case of disagreement, the opinion of a third reviewer was sought to reach a consensus. PROBAST categorises potential biases in prediction model studies into four domains: study population, predictors, outcomes and analysis, comprising a total of 20 signalling questions. Each signalling question can be answered by ‘yes/probably yes’, ‘no/probably no’ or ‘no information’. ‘Yes’ indicates a low risk of bias, ‘no’ indicates a high risk of bias and ‘no information’ indicates unclear risk. If at least one signalling question in a domain is answered by ‘no/probably no’, the domain is considered to have a high risk of bias. If any domain is assessed as having a high risk of bias, or if all four domains are assessed as having low risk of bias but the prediction model development lacks external validation, the study is classified as having a high risk of bias.

Patient and public involvement

None.

Results

This review retrieved a total of 4024 relevant studies. Following multiple rounds of screening to exclude literature not aligned with the study’s focus and duplicates, 3996 studies were removed, leaving 28 studies for final inclusion.^7–34 The detailed flow diagram of the literature screening process is presented in online supplemental material 2.

Summary of included studies

Among the 28 studies included in this review, publications within the past 5 years accounted for 42.9% (n=12). Of these, 14 were from the USA, 6 from China, 2 each from the UK and Canada, and 1 each from Japan, Australia and Italy. Additionally, there was a collaborative publication involving the Netherlands, Germany, Portugal, Austria and Belgium. Five studies were prospective,^{9 10 29 30 33} while 23 were multicentre studies.^{8 11–16 18–29 31–34} 19 studies sourced their data from registered databases,^{8 11–15 17–19 21 23–28 31 32 34} with 7 using the American College of Cardiology National Cardiovascular Data Registry and 4 using the New York State Percutaneous Coronary Intervention Reporting System. 10 studies focused exclusively on high-risk populations such as those with ST-elevation myocardial infarction (STEMI) and cardiogenic shock, the elderly and those on intensive care unit admissions.^{7 9 15–17 20 22 24 25} The primary outcome for seven studies was 30-day postoperative mortality,^{8 10 12 15 23 24 28} while one study assessed in-hospital and 30-day postoperative mortality.³² Basic characteristics of the included literature are presented in table 1.

Table 1

Characteristics of the included studies

Study	Year	Country	Design	Indication	Outcome	Mortality (T/V)
Li et al 7	2022	China	1	ACS	I	3.30%/3.70%
McAllister et al 8	2016	EnglandM	1	ACS/stable angina	II	1.70%/2.09%
Chiostri et al 9	2010	Italy	2	ACS	I	4.30%/3.83%
Song et al 10	2022	China	2	ACS/stable angina	II	0.24%/—
Wu et al 11	2006	USAM	1	—	I	0.70%/0.58%
Doll et al 12	2021	USAM	1	ACS/stable angina	II	1.66%/1.80%
Peterson et al 13	2010	USAM	1	ACS	I	1.24%/1.27%
Al’Aref et al 14	2019	USAM	1	—	I	—/—
Curtis et al 15	2012	USAM	1	ACS/stable angina	II	9.20%/9.00%
					II	1.40%/1.40%
Wang et al 16	2021	ChinaM	1	ACS	I	33.3%/19.8%
Yuan et al 17	2023	China	1	—	I	—/—
Shaw et al 18	2002	USAM	1	—	I	1.41%/1.43%
Brennan et al 19	2013	USAM	1	—	I	1.38%/1.40%
Gao et al 20	2020	ChinaM	1	ACS	I	8.10%/7.90%
de Mulder et al 21	2011	Netherlands et al M	1	ACS/stable angina	I	1.50%/1.30%
					I	5.4%/5.1%
Wang et al 22	2022	ChinaM	1	ACS	I	67.1%/55.8%
Hamburger et al 23	2009	CanadaM	1	ACS/stable angina	II	1.50%/1.40%
Cockburn et al 24	2020	EnglandM	1	ACS/stable angina	II	—/—
Klein et al 25	2002	USAM	1	ACS/stable angina	I	3.77%/—
Castro-Dominguez et al 26	2021	USAM	1	ACS	I	1.91%/1.93%
Inohara et al 27	2019	JapanM	1	ACS/stable angina/SMI/OMI	I	—/—
Tacey et al 28	2019	AustraliaM	1	ACS/stable angina	II	2.20%/—
Moscucci et al 29	2001	USAM	2	—	I	1.57%/1.70%
Qureshi et al 30	2003	USA	2	—	I	1.37%/1.01%
Burjonroppa et al 31	2011	USAM	1	ACS	I	—/—
Hannan et al 32	2013	USAM	1	ACS/stable angina	I/II	1.03%/—
Chowdhary et al 33	2009	CanadaM	2	—	I	1.30%/1.30%
Negassa et al 34	2007	USAM	1	ACS	I	3.60%/3.00%

1: retrospective cohort; 2: prospective cohort; I: death in hospital; II: death within 30 days; T:training; V:validation.

ACS, acute coronary syndrome; M, multicentre study; OMI, old myocardial infarction; SMI, silent myocardial infarction; T/V, training/validation.

Model construction and presentation

The 28 studies included in this review encompass a total of 38 prediction models. The number of candidate predictors in the included studies ranges from 8 to 80, with construction sample sizes ranging from 231 to 526 350 cases. The mortality rates vary from 0.24% to 67.1%, and the events per variable (EPV) range from 0.46 to 572.2. In terms of model construction methods, 24 studies employed the logistic regression approach,^{7 8 10 11 13 15–33} while 3 studies used machine learning algorithms such as AdaBoost and XGBoost.^{12 14 34} Among these, nine studies maintained the continuity of continuous variables,^{7 9 12 14 16 17 20 22 27} whereas six studies employed multiple imputation or complete case analysis to address missing data.^{14 17 18 21 24 25} In terms of model presentation, 11 prediction models were presented in the form of risk scoring systems, 7 prediction models were depicted through nomograms, 2 prediction models were presented as logistic regression equations, 2 prediction models were shown in the form of online calculators, 1 prediction model was displayed as a risk-stratified system and 1 prediction model was presented using a tree-structured risk stratification formula. Detailed information on model construction can be found in tables 2 and 3.

Table 2

Information on the construction of prediction models

Study	Sample size (T/V)	Number of events/EPV	Missing data number/processing	Construction methods	Model presentation
Li et al 7	480/108	16/0.46	—/—	LR	Logistic regression equation
McAllister et al 8	336 443/75 234	5722/572.2	—/median imputation/deleted	LR	Logistic regression equation
Chiostri et al 9	558/183	24/—	—/—	DA	–
Song et al 10	10 444/1000	25/1.09	—/—	LR	Nomogram
Wu et al 11	46 090/50 046	321/—	—/—	LR	Scoring system
Doll et al 12	46 907/11 727	—/—	—/—	LR/ML	Online calculator
Peterson et al 13	181 775/121 183	—/—	—/median imputation	LR	–
			—/—		Scoring system
Al’Aref et al 14	383 843/95 961	—/—	—/multiple imputation	LR/ML	–
Curtis et al 15	15 123/12 052	—/—	—/median imputation	LR	–
	110 529/88 630				–
Wang et al 16	396/459	132/4	—/—	LR	Nomogram
Yuan et al 17	1296/864	—/—	—/multiple imputation/deleted	LR	Nomogram
Shaw et al 18	50 123/50 130	707/22.09	—/multiple imputation	LR	–
					–
					–
Brennan et al 19	724 883/483 254	—/—	—/median imputation	LR	–
					–
					Scoring system
Gao et al 20	1169/316	95/1.19	—/—	LR	Nomogram
de Mulder et al 21	23 032/23 032	339/—	—/multiple imputation	LR	Scoring system
	4091/3969				Scoring system
Wang et al 22	231/43	155/5.17	—/—	LR	Nomogram
Hamburger et al 23	526 350/56 549	406/—	2640/deleted	LR	Online calculator
Cockburn et al 24	22 072/22 071	—/—	—/multiple imputation/complete case analysis	LR	Nomogram
Klein et al 25	8828/—	333/—	—/multiple imputation	LR	Nomogram
Castro-Dominguez et al 26	495 005/211 258	—/—	—/median imputation	LR	–
					–
					Scoring system
Inohara et al 27	334 591/334 590	—/—		LR	–
					Scoring system
Tacey et al 28	19 258/8254	431/15.39	—/median imputation	LR	–
Moscucci et al 29	10 729/5863	—/—	—/median imputation	LR	Scoring system
Qureshi et al 30	9954/12 005	—/—	—/—	LR	Risk-stratified system
Burjonroppa et al 31	5569/2438	—/—	—/median imputation	LR	Scoring system
Hannan et al 32	54 223/53 893	—/—	—/—	LR	Scoring system
Chowdhary et al 33	10 694/5347	139/9.27	—/median imputation	LR	Scoring system
Negassa et al 34	5385/7414	—/—	—/—	ML	Tree-structured risk stratification formula

DA, discriminant analysis; EPV, events per variable; LR, logistic regression; ML, machine learning; T, training; V, validation.

Table 3

Information on the performance of prediction models

Study	Internal validation (IV)	External validation (EV)	AUC (T)	AUC (V)	Calibration
Li et al 7	—	EV1	0.938	0.937	HL
McAllister et al 8	—	EV1	0.85	0.86	CC
Chiostri et al 9	—	EV1	0.986	0.968	—
Song et al 10	IV2		0.881	0.884	CC
Wu et al 11	—	EV1	0.886	0.905	CC
Doll et al 12	IV1	EV1/EV2	0.92	0.86	CC
Peterson et al 13	IV1	EV1	—	0.924	CC
			—	0.905	CC
Al’Aref et al 14	IV1/IV3	—	0.927	—	CC
Curtis et al 15	—	EV1	0.83	0.84	CC
			0.82	0.81	CC
Wang et al 16	—	EV1	0.947	0.891	CC
Yuan et al 17	IV1	—	0.907	0.901	CC
Shaw et al 18	IV1	—	0.89	0.89	CC
			0.87	—	HL
			0.86	—	HL
Brennan et al 19	IV1	—	0.931	0.93	CC
			0.929	0.929	—
			0.925	0.925	—
Gao et al 20	—	EV1	0.987	0.990	CC
de Mulder et al 21	IV1	—	0.91	0.9	CC
			0.86	0.89	HL
Wang et al 22	—	—	0.819	0.842	CC
Hamburger et al 23	—	EV1	0.902	0.911	HL
Cockburn et al 24	IV1	—	0.83	—	—
Klein et al 25	—	—	0.829	—	CC
Castro-Dominguez et al 26	IV1	—	0.943	0.943	CC
			0.94	0.94	CC
			0.924	0.924	CC
Inohara et al 27	IV1	—	—	0.929	CC
			—	0.926	—
Tacey et al 28	IV1	EV1	0.921	0.934	CC
Moscucci et al 29	IV2	EV1	0.90	0.92	CC
Qureshi et al 30	—	EV1	0.81	0.825	HL
Burjonroppa et al 31	IV1	—	0.88	0.88	CC
Hannan et al 32	—	EV1	0.89	—	CC
Chowdhary et al 33	IV1	—	—	0.96	CC
Negassa et al 34	—	EV1	—	0.82	—

IV1: randomised splitting method; IV2: bootstrap resampling; IV3: cross-validation; EV1: temporal validation; EV2: geographical validation.

AUC, area under the curve; CC, calibration curve; HL, Hosmer-Lemeshow; T, training; V, validation.

Model performance and predictors

Among the 28 studies included, 3 studies employed internal validation using cross-validation or bootstrap resampling methods,^{10 14 29} while 15 studies used external validation methods, including temporal and geographical validation.^{7–9 11–13 15 16 20 23 28–30 32 34} The area under the receiver operating characteristic curve (AUC) during model development ranged from 0.81 to 0.987, while during model validation, it ranged from 0.81 to 0.99. 22 studies provided calibration curve plots. The number of predictors included in the models ranged from 3 to 21, with the most common predictors were age, shock, left ventricular ejection fraction, site of myocardial infarction and elevation of ST segment on ECG.^{9–22 25–29 31–33} The top 20 most frequently occurring predictors are depicted in online supplemental material 3.

Risk of bias and applicability assessment

Except for the study by Al’Aref et al,¹⁴ whose risk of bias is unclear, the remaining 27 studies all exhibited a high risk of bias, mainly in the domains of prediction and analysis. Specific evaluation results are shown in online supplemental material 4. The reasons for biases and the limitations of the studies are detailed in online supplemental material 5. Additionally, specific assessment details using PROBAST are provided in online supplemental material 6.

Domain 1: participants

Six studies exhibited a high risk of bias. Among them, four studies used data from retrospective cohort studies.^{7 16 20 22} The remaining two studies, although based on registry data, employed inappropriate participant selection strategies, including participants with a higher risk of mortality.^{15 31}

Domain 2: predictors

Three studies exhibited a high risk of bias due to the use of multicentre data for retrospective analysis^{16 20 22}; 5 prospective cohort studies were deemed to show a low risk of bias as they inherently employed blinding^{9 10 29 30 33}; and the remaining 20 studies were identified with an unclear risk of bias as they did not report the methods used to measure or assess the predictors.

Domain 3: outcome

All 28 studies failed to clearly define their outcome indicators and measurement methods, and were therefore assessed as having an unclear risk of bias. Although death is a definitive clinical endpoint that typically does not involve subjective judgement or significant measurement errors, this review encompasses studies from multiple countries and regions, where standards for recording and confirming death may vary. Moreover, this review includes studies predicting mortality risk in patients with severe and terminal illnesses. In such cases, clearly describing the definitions, causes and methods for documenting death outcomes is crucial for interpreting the results. Thus, all 28 studies were assessed as having an unclear risk of bias.

Domain 4: analysis

The study by Al’Aref et al did not report the methods used to handle missing values and sample size information, and the evaluation results for other signalling questions were marked as ‘yes/probably yes’. Thus, it was assessed as having an unclear risk of bias.¹⁴ The remaining 27 studies were assessed as having a high risk of bias. Among these, 5 studies had an EPV count of less than 10^{7 16 20 22 33}; 18 studies categorised continuous variables such as age and glomerular filtration rate as discrete variables^{7 8 11 13 15 18 19 21 23–26 28–30 32–34}; 2 studies excluded some participants from the statistical analysis^{12 23}; 11 studies resorted to inappropriate methods like median imputation to handle missing data^{8 13 15 17 19 23 26 28 29 31 33}; 17 studies chose predictors solely based on univariate analysis^{9 10 13 16 18 19 21–26 28–30 32 33}; 5 studies failed to specify the exclusion of significant competing risks of death due to non-cardiac causes, overlooking the complexity of the data^{11 25 31 32 34}; 3 studies relied only on the Hosmer-Lemeshow goodness-of-fit test for calibration without reporting calibration plots or tables^{7 23 30}; 10 studies conducted internal validation through random data splitting^{13 17–19 21 24 26–28 31}; 4 studies did not refit predictor coefficients after selecting relevant predictors from multivariate regression analysis^{22 24 31 33}; and 2 studies reported inconsistencies between predictors and their regression coefficients.^{30 34}

Applicability

27 studies exhibited a high applicability risk. Of these, 10 focused on critically ill or elderly patients with high mortality rates,^{7 9 15–17 20 22 24 25 31} and 21 required the evaluation of predictive factors during or after PCI surgery.^{7 9–11 13–16 18–23 26–30 32 33}

Discussion

Model performance and risk of bias need improvement

All models constructed in the studies included in this review demonstrated good predictive performance (AUC >0.75) and effectively identified patients at high risk of short-term mortality after PCI. Among them, the model by Gao et al showed the best predictive performance after external validation.²⁰ However, most studies were assessed as having a high risk of bias, with only one study having an unclear overall risk of bias.

In terms of study design, four retrospective cohort studies were assessed as having a high risk of bias due to potential data omissions, input errors and differences in evaluation methods, which could affect data quality and integrity. It is recommended that model construction prioritise prospective study data to improve data quality and reduce bias and errors.³⁵ Two studies that included high-risk patients with high mortality rates were assessed as having a high risk of bias because their predictive models might overly rely on these patients’ specific high-risk characteristics, neglecting other important predictors for the overall patient population. This could lead to model overfitting and overestimation of mortality risk for the general patient population. Therefore, study samples should have broad representation, including patients of varying risk levels, to enhance model generalisability and clinical applicability. Three studies using multicentre data were assessed as having a high risk of bias due to potential significant data bias arising from different data collection standards across centres. Thus, before conducting multicentre studies, researchers should establish clear research protocols and standardise data collection and statistical procedures to reduce risk of bias and enhance the scientific value of the research.³⁶

In model construction, five studies were assessed as having a high risk of bias due to an EPV of less than 10. A low EPV can lead to overestimation or underestimation of predictor variable effects, increasing statistical instability. To improve accuracy and stability, it is generally recommended ensuring an EPV of at least 20.³⁷ 18 studies were assessed as having a high risk of bias for categorising continuous variables as discrete. This simplification can result in the loss of crucial information and distort relationships between variables, affecting model accuracy; thus, unnecessary discretisation of continuous variables should be avoided. If categorisation is necessary, appropriate thresholds based on clinical and statistical needs should be selected and using multicategory or ordinal classifications to retain more information should be considered. Two studies introduced bias by excluding certain participants, which could alter the distribution of key variables and create differences between the remaining sample and the overall patient population. In this case, excluding participants without sufficient justification should be avoided. If exclusion is necessary, thoroughly assessing and reporting the potential bias impact on study results are important. 11 studies were assessed as having a high risk of bias due to improper handling of missing data, which could lead to information loss or the introduction of misleading data, distorting prediction outcomes. Advanced techniques like multiple imputation should be used to handle missing data, rather than using simple single-value replacements.³⁸ 17 studies were assessed as having a high risk of bias for selecting predictors solely based on univariate analysis. This method ignores interactions between variables and the complex relationships between predictors and outcomes, such as non-linear relationships and threshold effects, leading to poor model performance in practical applications. Use of multivariate analysis techniques to consider these interactions and complexities is necessary, ensuring the model is built on multidimensional variable effects. Five studies did not explicitly exclude non-cardiac death as a competing risk, resulting in high risk of bias. This omission can lead to external deaths being mistakenly attributed to cardiac causes, obscuring the true impact of cardiac-related risk factors and leading to inaccurate estimates of cardiac death risk.³⁹ Excluding all relevant competing risks during model development is crucial. Four studies were assessed as having a high risk of bias for not recalibrating predictor coefficients after selection in multivariate regression analysis. This can overlook interactions between variables, preventing the model from accurately reflecting the true effects of the predictors and reducing predictive ability. Recalibrating predictor coefficients after multivariate regression analysis to ensure model accuracy and stability is necessary to reduce risk of bias.

In terms of model validation and presentation, three studies were assessed as having a high risk of bias for not reporting calibration plots or tables. This omission prevents the assessment of calibration performance across different risk levels and may overlook calibration issues.⁴⁰ Calibration plots or tables provide visual information that helps identify calibration problems within specific probability ranges, detect trends and systematic errors, increase transparency, and offer a basis for comprehensive model evaluation and improvement. Therefore, it is recommended providing calibration plots or tables in addition to using the Hosmer-Lemeshow test when calibrating models. 10 studies were assessed as having a high risk of bias due to using random data splitting for internal validation. Random splitting can lead to data unevenness between the training and validation sets or data leakage, affecting model stability and generalisability. To improve model reliability and reduce risk of bias, it is recommended using more robust internal validation methods, such as cross-validation (eg, k-fold cross-validation) or leave-one-out validation, combined with external validation covering diverse populations and clinical settings, to reflect the model’s feasibility in actual clinical practice.⁴¹ Additionally, ensuring sample independence in the construction and validation of different models is crucial to enhance the applicability of models across different patient groups and avoid bias from duplicate samples. Two studies were assessed as having a high risk of bias due to inconsistencies between reported predictors and their regression coefficients, reflecting potential bias in the variable selection process, possibly due to overfitting or improper statistical handling. These inconsistencies can reduce the interpretability of the model, making the study results difficult to understand and apply. To avoid this risk of bias, ensuring data quality is important, as well as transparently reporting the variable selection process and regression coefficient calculations, making the research process reproducible for other researchers. Furthermore, it is essential to regularly update and refine models as new data and technologies become available.

Predictors are subject to further discussion

The predictive factors included in this study vary due to factors such as data source availability and the timing of model development. Key predictors include age, cardiogenic shock, low ejection fraction, high-risk myocardial infarction sites and elevated ST segments on ECGs. These indicators collectively suggest a preoperative state of reduced patient tolerance and significant cardiovascular impairment, which increases the risk of surgical complications and elevates short-term postoperative mortality.⁴² This aligns with findings from Komorova et al’s research.⁴³ Consequently, these predictors highlight the need for clinicians to rigorously assess cardiovascular status perioperatively, adhere to the latest clinical guidelines and best practices, and strictly manage surgical indications. Other studies indicate that mechanical complications of myocardial infarction, such as ventricular septal defects, lead to high mortality rates unaffected by reperfusion therapy.⁴⁴ Future research should focus on evaluating their impact on the accuracy of postoperative mortality risk predictions. Furthermore, since some predictors in 21 studies require assessment during or after surgery, they are unable to assist in preoperative planning. To better meet diverse risk prediction needs, future research should focus on developing precise models tailored to different clinical stages, as exemplified by several models developed by Peterson and his team.^{13 19 26 27}

Prospects for model construction

In this review, only three models included were based on machine learning algorithms. Among them, Doll et al employed machine learning algorithms for variable reduction.¹² Al’Aref et al compared the predictive performance of various machine learning algorithms with traditional logistic regression,¹⁴ revealing that AdaBoost and XGBoost outperformed logistic regression. However, the decision tree model constructed by Negassa et al exhibited relatively average predictive performance compared with other logistic regression models due to the inclusion of fewer predictive variables.³⁴ In recent years, machine learning algorithms have been widely applied and developed in data processing, model construction and performance evaluation. Relevant studies have shown that they can effectively handle high-dimensional, non-linear and complex data, thereby improving model accuracy and generalisation ability.⁴⁵ Although the predictive performance of machine learning algorithms in existing studies is generally moderate, future research should focus on addressing challenges such as algorithm bias and model interpretability, fully exploring their potential and value in constructing clinical risk prediction models.

10 studies included in this review constructed models based on datasets of high-risk patients, such as those with STEMI or cardiogenic shock, where the postoperative mortality rate of these patients is significantly increased. While this enhances the predictive accuracy of the models for specific populations, it also compromises their generalisability in clinical practice. Clinicians should integrate both specific and generalisable models to provide comprehensive prediction and decision support.⁴⁶ Before application, it is essential to carefully consider the scope and limitations of the models to avoid indiscriminate generalisation to patient populations different from the development dataset. Additionally, efforts can be made to develop dynamic risk prediction models by incorporating changes in information from patient electronic health records in real time or periodically, enabling early identification of high-risk populations and achieving continuous, comprehensive risk monitoring throughout the entire care continuum.⁴⁷ This will provide timely information for clinical care teams to improve patient outcomes and reduce the incidence of adverse events.⁴⁸

Limitations

This review has certain limitations: first, only Chinese and English literature was included, potentially omitting studies in other languages and causing some publication bias. Second, due to differences in sample characteristics, variable selection and construction algorithms among different modelling studies, only qualitative analysis was conducted, and a meta-analysis was not feasible. Third, the included machine learning algorithm models involve complex algorithms and data processing procedures, which may have bias risks that cannot be fully assessed by PROBAST. Fourth, sample overlap in seven studies could overemphasise characteristics specific to that dataset, possibly limiting the models’ applicability and introducing bias into the findings.

Conclusion

This review analysed 38 models predicting in-hospital and 30-day mortality risk after PCI. While existing models show good predictive performance, they are often associated with high risk of bias, which raises concerns about their clinical applicability and prediction stability. Future model construction or validation research should strictly adhere to the low risk of bias criteria of relevant items in PROBAST. Moreover, researchers should also attempt to correctly apply machine learning algorithms to develop prediction models with ideal performance and clinical applicability, while comprehensively considering clinical high mortality risk factors. Furthermore, it is crucial to emphasise the external validation of these models post-construction to ensure their applicability across diverse clinical datasets and varying distributions.

This work is supported by the Zhejiang Traditional Chinese Medicine Inheritance and Innovation Project(2023ZX0950)，the First-Class Discipline Project of Zhejiang Province(1133) and Medical and Health Technology Plan of Zhejiang Province (2022507615).

We would like to thank the Coronary Care Unit and the Digital Subtraction Angiography Unit of Zhejiang Provincial People's Hospital for their support in this review.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information. All relevant data for the study are included in the article or provided as supplemental material.

Ethics statements

Patient consent for publication

Not applicable.

Ethics approval

Not applicable.

Footnote

YS and CZ contributed equally.

Contributors As the first author of this review, YS was responsible for overall project management, risk of bias assessment, analysis of evaluation results and writing the paper. Co-first author CZ was in charge of literature screening, risk of bias assessment and revising the paper. WQ primarily handles literature screening. SC was responsible for arbitrating literature screening results and paper submission, and acts as the guarantor, ensuring the integrity of the work as a whole. XC arbitrated data extraction results. DX and CW were responsible for extracting and verifying the data from the included literature.

Funding This review was supported by the Chinese Medical Association Journal Research Program Fund (CMAPH-NRG2021035).

Competing interests None declared.

Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.

Provenance and peer review Not commissioned; externally peer reviewed.

Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.

References

¹ SHENGSHOU. Report on cardiovascular health and diseases in China 2022: an updated Summary. Biomed Environ Sci 2023; 36: 669–701.

² SHOUWEI Z, LONGSHAN Y, ZONGBAO H. Construction and verification of the risk scoring criteria for ventricular arrhythmia after emergency PCI in elderly patients with acute myocardial Infarction. Chinese J Crit Care Med 2023; 43: 734–40.

³ Zi-Xuan L, Qi-Qi C, Wan-Li X. The trend on the Interventional diagnosis and treatment of coronary heart disease in Hubei province from 2016 to 2021. Chinese J Intervent Cardiol 2023; 31: 286–91.

⁴ Kim M, Kang D, Kim MS, et al. Acute myocardial infarction prognosis prediction with reliable and interpretable artificial intelligence System. J Am Med Inform Associat 2024; 31: 1540–50. doi:10.1093/jamia/ocae114

⁵ Mei W, Qifeng G, Linan D. Association between Subclinical hypothyroidism and death in elderly patients after percutaneous coronary Intervention. Pract Geriat 2023; 37: 138–41.

⁶ Xinyi Z, Ranxun A, Jianhua S. Research progress on risk prediction model of central venous catheter-related thrombosis. Chinese Nurs Manag 2023; 23: 1592–5.

⁷ Li Q, Yi Q, Xv G. Establishment and validation of prediction model of in-hospital mortality risk in patients with acute ST- segment elevation myocardial infarction after primary percutaneous coronary intervention. Chin J Evid Based Cardiovasc Med 2022; 14: 1470–5.

⁸ McAllister KSL, Ludman PF, Hulme W, et al. A contemporary risk model for predicting 30-day mortality following percutaneous coronary intervention in England and Wales. Int J Cardiol 2016; 210: 125–32. doi:10.1016/j.ijcard.2016.02.085

⁹ Chiostri M, Valente S, Crudeli E, et al. A new post-PCI scoring system for in-hospital mortality in STEMI patients. J Cardiovasc Med 2010; 11: 733–8. doi:10.2459/JCM.0b013e328339d910

¹⁰ Song J, Liu Y, Wang W, et al. A nomogram predicting 30-day mortality in patients undergoing percutaneous coronary intervention. Front Cardiovasc Med 2022; 9: 897020. doi:10.3389/fcvm.2022.897020

¹¹ Wu C, Hannan EL, Walford G, et al. A risk score to predict in-hospital mortality for percutaneous coronary interventions. J Am Coll Cardiol 2006; 47: 654–60. doi:10.1016/j.jacc.2005.09.071

¹² Doll JA, O’Donnell CI, Plomondon ME, et al. Contemporary clinical and coronary anatomic risk model for 30-day mortality after percutaneous coronary intervention. Circ: Cardiovas Intervent 2021; 14. doi:10.1161/CIRCINTERVENTIONS.121.010863

¹³ Peterson ED, Dai D, DeLong ER, et al. Contemporary mortality risk prediction for percutaneous coronary intervention: results from 588,398 procedures in the National cardiovascular data Registry. J Am Coll Cardiol 2010; 55: 1923–32. doi:10.1016/j.jacc.2010.02.005

¹⁴ Al’Aref SJ, Singh G, van Rosendael AR, et al. Determinants of in-hospital mortality after percutaneous coronary intervention: A machine learning Approach. J Am Heart Assoc 2019; 8: e011160. doi:10.1161/JAHA.118.011160

¹⁵ Curtis JP, Geary LL, Wang Y, et al. Development of 2 Registry-based risk models suitable for characterizing hospital performance on 30-day all-cause mortality rates among patients undergoing percutaneous coronary Intervention. Circ: Cardiovas Qual Outcomes 2012; 5: 628–37. doi:10.1161/CIRCOUTCOMES.111.964569

¹⁶ Wang Y, Wang W, Jia S, et al. Development of a Nomogram for the prediction of in-hospital mortality in patients with acute ST-elevation myocardial infarction after primary percutaneous coronary intervention: a Multicentre, retrospective, observational study in Hebei province, China. BMJ Open 2022; 12: e056101. doi:10.1136/bmjopen-2021-056101

¹⁷ Yuan M, Ren BC, Wang Y, et al. Development of a novel tool: a Nomogram for predicting in-hospital mortality of patients in intensive care unit after percutaneous coronary Intervention. BMC Anesthesiol 2023; 23: 5. doi:10.1186/s12871-022-01923-y

¹⁸ Shaw RE, Anderson HV, Brindis RG, et al. Development of a risk adjustment mortality model using the American college of cardiology-national cardiovascular data Registry (ACC-NCDR) experience: 1998-2000. J Am Coll Cardiol 2002; 39: 1104–12. doi:10.1016/s0735-1097(02)01731-x

¹⁹ Brennan JM, Curtis JP, Dai D, et al. Enhanced mortality risk prediction with a focus on high-risk percutaneous coronary intervention: results from 1,208,137 procedures in the NCDR (national cardiovascular data Registry). JACC Cardiovasc Interv 2013; 6: 790–9. doi:10.1016/j.jcin.2013.03.020

²⁰ Gao N, Qi X, Dang Y, et al. Establishment and validation of a risk model for prediction of in-hospital mortality in patients with acute ST-elevation myocardial infarction after primary PCI. BMC Cardiovasc Disord 2020; 20: 513. doi:10.1186/s12872-020-01804-7

²¹ de Mulder M, Gitt A, van Domburg R, et al. Euroheart score for the evaluation of in-hospital mortality in patients undergoing percutaneous coronary Intervention. Europ Heart J 2011; 32: 1398–408. doi:10.1093/eurheartj/ehr034

²² Wang Y, Liu L, Li X, et al. Nomogram for predicting in-hospital mortality in patients with acute ST-elevation myocardial infarction complicated by cardiogenic shock after primary percutaneous coronary Intervention. J Interv Cardiol 2022; 2022: 8994106. doi:10.1155/2022/8994106

⁴ Hamburger JN, Walsh SJ, Khurana R, et al. Percutaneous coronary intervention and 30‐Day mortality: the British Columbia PCI risk score. Cathet Cardio Intervent 2009; 74: 377–85. doi:10.1002/ccd.22151

²⁴ Cockburn J, Kemp T, Ludman P, et al. Percutaneous coronary intervention in Octogenarians: A risk scoring system to predict 30-day outcomes in the Elderly. Cathet Cardio Intervent 2021; 98: 1300–7. doi:10.1002/ccd.29406

²⁵ Klein LW, Block P, Brindis RG, et al. Percutaneous coronary interventions in Octogenarians in the American college of cardiology-national cardiovascular data Registry - development of a Nomogram predictive of in-hospital mortality. J Am Coll Cardiol 2002; 40: 394–402. doi:10.1016/s0735-1097(02)01992-7

²⁶ Castro-Dominguez YS, Wang Y, Minges KE, et al. Predicting in-hospital mortality in patients undergoing percutaneous coronary Intervention. J Am Coll Cardiol 2021; 78: 216–29. doi:10.1016/j.jacc.2021.04.067

²⁷ Inohara T, Kohsaka S, Yamaji K, et al. Risk stratification model for in-hospital death in patients undergoing percutaneous coronary intervention: a nationwide retrospective cohort study in Japan. BMJ Open 2019; 9: e026683. doi:10.1136/bmjopen-2018-026683

²⁸ Tacey M, Dinh DT, Andrianopoulos N, et al. Risk-adjusting key outcome measures in a clinical quality PCI Registry. JACC: Cardiovascular Interventions 2019; 12: 1966–75. doi:10.1016/j.jcin.2019.07.002

²⁹ Moscucci M, Kline-Rogers E, Share D, et al. Simple bedside additive tool for prediction of in-hospital mortality after percutaneous coronary Interventions. Circulation 2001; 104: 263–8. doi:10.1161/01.CIR.104.3.263

³⁰ Qureshi MA, Safian RD, Grines CL, et al. Simplified scoring system for predicting mortality after percutaneous coronary Intervention. J Am College Cardiol 2003; 42: 1890–5. doi:10.1016/j.jacc.2003.06.014

³¹ Burjonroppa SC, Varosy PD, Rao SV, et al. Survival of patients undergoing rescue percutaneous coronary Interve. JACC: Cardiovascular Interventions 2011; 4: 42–50. doi:10.1016/j.jcin.2010.09.020

³² Hannan EL, Farrell LS, Walford G, et al. The New York State risk score for predicting in-Hospital/30-day mortality following percutaneous coronary intervention. JACC Cardiovasc Interv 2013; 6: 614–22. doi:10.1016/j.jcin.2013.02.015

³³ Chowdhary S, Ivanov J, Mackie K, et al. The Toronto score for in-hospital mortality after percutaneous coronary interventions. Am Heart J 2009; 157: 156–63. doi:10.1016/j.ahj.2008.08.026

³⁴ Negassa A, Monrad ES, Bang JY, et al. Tree-structured risk stratification of in-hospital mortality after percutaneous coronary intervention for acute myocardial infarction: a report from the new york state percutaneous coronary intervention database. Am Heart J 2007; 154: 322–9. doi:10.1016/j.ahj.2007.03.052

³⁵ Wang S, Huang H, Hou M, et al. Risk-prediction models for intravenous immunoglobulin resistance in Kawasaki disease: risk-of-bias assessment using PROBAST. Pediatr Res 2023; 94: 1125–35. doi:10.1038/s41390-023-02558-6

³⁶ Hedberg P, Varisco B, Bai F, et al. Eucare-hospitalised study protocol: a cohort study of patients hospitalised with COVID-19 in the Eucare project. BMC Infect Dis 2023; 23: 690. doi:10.1186/s12879-023-08658-2

³⁷ Moons KGM, Wolff RF, Riley RD, et al. PROBAST: A tool to assess risk of bias and applicability of prediction model studies: explanation and elaboration. Ann Intern Med 2019; 170: W1–33. doi:10.7326/M18-1377

³⁸ Janssen KJM, Donders ART, Harrell FE, et al. Missing Covariate data in medical research: to impute is better than to Ignore. J Clin Epidemiol 2010; 63: 721–7. doi:10.1016/j.jclinepi.2009.12.008

³⁹ SCORE2 working group and ESC Cardiovascular risk collaboration. Score2 risk prediction Algorithms: new models to estimate 10-year risk of cardiovascular disease in Europe. Eur Heart J 2021; 42: 2439–54. doi:10.1093/eurheartj/ehab309 34120177

⁴⁰ Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for Validation. Eur Heart J 2014; 35: 1925–31. doi:10.1093/eurheartj/ehu207

⁴¹ Kunonga TP, Kenny RPW, Astin M, et al. Predictive accuracy of risk prediction models for recurrence, metastasis and survival for early-stage cutaneous Melanoma: a systematic review. BMJ Open 2023; 13: e073306. doi:10.1136/bmjopen-2023-073306

⁴² Zong M, Guan X, Chang J, et al. Effect of frailty on elderly patients suffering from heart failure with reduced ejection fraction. Chin J Geriatr Heart Brain Ves Dis 2024; 26: 523–6.

⁴³ Komorova Z, Machal J, Groch L, et al. Effect of selected cardiovascular factors on mortality in patients with ST-segment elevation myocardial infarction treated with primary coronary Intervention. BLL 2023; 124: 421–6. doi:10.4149/BLL_2023_064

⁴⁴ Ouaddi NE, de Diego O, Labata C, et al. Mechanical complications in STEMI: prevalence and mortality trends in the primary PCI era. The Ruti-STEMI Registry. Revista Española de Cardiología (English Edition) 2023; 76: 427–33. doi:10.1016/j.rec.2022.09.012

⁴⁵ Xenhui J, Qi L, Jing P. Research status on methodology application of readmission risk prediction model for patients with acute myocardial infarction. Chinese Nurs Res 2023; 37: 3111–5.

⁴⁶ Huang L, Mao D, Zheng J, et al. Application of artificial intelligence in HE risk prediction modelling and research advances. J Pract Med 2024; 40: 289–94.

⁴⁷ Heyard R, Timsit J, Essaied WI, et al. Dynamic clinical prediction models for discrete time-to-event data with competing risks-A case study on the OUTCOMEREA database. Biometrical J 2019; 61: 514–34. doi:10.1002/bimj.201700259

⁴⁸ Jie Z, Hong C, LI P. A review of Wearable devices in the monitoring of stroke risk factors and risk prediction. Chinese J Nurs 2022; 57: 1141.