Impact of type of procedure and surgeon on EuroSCORE operative risk validation

Abstract

Objective: EuroSCORE has been used in cardiac surgery operative risk assessment, despite important variables were not included. The objective of this study was to validate EuroSCORE on mortality prediction in a Brazilian cardiovascular surgery center, defining the influence of type of procedure and surgical team. Methods: Between January 2006 and June 2011, 2320 consecutive adult patients were studied. According to additive EuroSCORE, patients were divided into low risk (score<2), medium risk (3 - 5), high risk (6 - 11) and very high risk (>12). The relation between observed mortality (O) and expected mortality (E) according to logistic EuroSCORE was calculated for each of the groups, types of procedures and surgeons with > 150 operations, and analyzed by logistic regression. Results: EuroSCORE correlated to the observed mortality (O/E=0.94; P<0.0001; area under the curve 0.78). However, it overestimated the mortality in very high risk patients (O/E=0.74; P=0.001). EuroSCORE tended to overestimate isolated myocardial revascularization mortality (O/E=0.81; P=0.0001) and valve surgery mortality (O/E=0.89; P=0.007) and it tended to underestimate combined procedures mortality (O/E=1.09; P<0.0001). EuroSCORE overestimated surgeon A mortality (O/E=0.46; P<0.0001) and underestimated surgeon B mortality (O/E=1.3; P<0.0001), in every risk category. Conclusion: In the present population, EuroSCORE overestimates mortality in very high risk patients, being influenced by type of procedure and surgical team. The most appropriate surgical team may minimize risks imposed by preoperative profiles.

INTRODUCTION

The scores of preoperative risk aim to estimate the mortality of certain surgical procedures in order to meet the interests of individuals (patients and physicians), hospitals and health managers. The risk prediction is important in medical practice, as it allows objective comparisons between institutions and surgeons to adjust the characteristics of severity of disease. Moreover, the scores of preoperative risk are useful in clarifying and preoperative consent, in quality control of services and the selection or exclusion of patients in controlled clinical studies.

The EuroSCORE (European System for Cardiac Operative Risk Evaluation), is one of the more widespread preoperative risk scores, and perhaps the most popular in cardiac surgery. It was created in 1999^[1] , with data collected between September and December 1995 in 128 European centers. In 2003, the logistic EuroSCORE was introduced using the same original database in order to improve the performance of the score in high risk patients^[2] .

External validation of the EuroSCORE in various populations around the world occurred initially with good results^[3-5] . However, most authors have recently demonstrated that the logistic EuroSCORE overestimates the expected mortality, although some developing countries show the opposite^[6-8] . The reasons for this discrepancy in model validation involve multiple factors: different characteristics of the exposed population, the hospitals and their treatment protocols, training teams and socioeconomic^[9] differences.

Moreover, the first version of the EuroSCORE did not take into account all types of surgical procedures used in cardiac surgery because the vast majority of patients underwent isolated CABG, which may lead to misinterpretation in various possibilities of combined procedures ^[10] .

The influence of the hospital unit or a system of national health outcome in cardiac surgery has been studied in this context^[11] . Siregar et al.^[12] showed the limitations of showing lists of performance of hospitals and surgeons in the study population of patients undergoing cardiovascular surgery in the Netherlands. The study shows that in a three-year period, there was great variability in the results of hospitals with wide confidence intervals, thus questioning the validity of such comparisons. The inaccuracy of mathematical models of risk, reflecting differences in severity of disease and risk factors not measured, may reflect results of random variations rather than real ones.

The aim of this study was to validate the EuroSCORE in predicting cardiovascular mortality in a Brazilian center surgery, defining the influence of the type of procedure and surgeon factor.

METHODS

From January 2006 to June 2011, 2320 consecutive adult patients who had undergone surgery were included in this study. The mean age was 55 years ± 15, and 1330 (57.3%) patients were male. The preoperative, intraoperative and postoperative patient characteristics were prospectively collected and stored in an electronic database. All patients underwent surgery in the same hospital following the same protocols. The study was approved by the Research Ethics Committee under the registration number 069882/2013, in accordance with the standards of Helsinki.

The procedures were coronary artery bypass grafting (CABG) isolated in 1056 (45.6%), valve surgery (Valve) isolated in 627 (27.1%), combined operations in 453 (19.6%) and others at rest. The latter included surgical treatment of isolated aortic diseases, congenital heart disease in adult, pericardiectomy, septal myectomy, resection of cardiac tumors, VSD repair or post-myocardial infarction heart failure and surgical treatment of endomyocardial fibrosis.

The primary outcome was mortality at 30 days. There was no loss of patient follow up at 30 days. The operative risk score used was the first version of the EuroSCORE, published in 1999^[1] . The calculation of the EuroSCORE was prospectively performed by a single examiner who strictly followed the definitions of pre- and intraoperative characteristics of patients. The calculation of the additive and logistic score of each patient was then determined electronically accessing http://www.EuroSCORE.org/calcold.html.

Table 1 defines the prevalence of variables used in calculating the score compared to the same used in the formation of the original statistical model.

Table 1 Prevalence of variables included in the calculation of the EuroSCORE compared with data from the original publication.

[ Table Omitted - see PDF ]

Statistical Analysis

Categorical variables were expressed by frequencies and percentages and continuous variables through the mean and standard deviation or with confidence interval of 95%. Comparison of categorical variables was performed using the chi-square test and continuous using Student's T test, as indicated. Validation of the EuroSCORE for the population studied was performed, including calibration and discrimination. In the latter analysis, all patients were included. The correlation between mortality and the risk score, type of procedure and surgical team was obtained by logistic regression, ROC (receiver-operating characteristic curve) was obtained and calculated the area under the curve (AUC). In logistic regression analysis, the variables whose P value was less than or equal to 0.1 were included in the multivariate model. The latter was performed using the backward stepwise technique, and the variables were selected with a P value less than 0.05 as significant. Statistical analyzes were performed using JMP software, version 9.0, SAS Institute.

RESULTS

In the population studied, the EuroSCORE calibrated with the observed mortality (O/E relationship 0.94, 95% CI 0.73 to 1.04, P <0.0001). The analysis of the ROC curve (Figure 1) revealed discriminatory low power (area under the curve = 0.78) of the EuroSCORE in predicting mortality when assessed all the study population. The sensitivity was 67.8% and specificity of 88%.

Fig. 1 Relationship between the logistic EuroSCORE and 30-day mortality after cardiovascular surgery using the ROC curve

The EuroSCORE> 7, and the cardiopulmonary bypass time greater than 120 minutes were identified as independent risk factors for mortality in this population, as shown in Table 2 which expresses the multivariate analysis of risk factors for mortality. Still, the surgeon A was inversely related to the primary event for the same model.

Table 2 Risk factors for 30-day mortality by multivariate logistic regression

[ Table Omitted - see PDF ]

Estimation of operative risk according to emergency operation

There was a higher mortality in emergency operations than in elective (33.8% vs . 5.3%, P <0.0001), but were consistent with the estimated risk by EuroSCORE. There was no difference in the surgeon factor (P =0.22) and in the type of procedure (P =0.11) with regard to mortality outcomes in emergency operations.

Estimation of operative risk according to risk groups (additive EuroSCORE)

As shown in Table 3, there were differences in estimating operative risk according to severity. The EuroSCORE is poorly calibrated for patients with moderate and high risk, and tended to overestimate mortality in patients at low risk (O/E = 0.65, 95% CI 0.29 to 1.44, P =0.006; AUC=0.6) despite the discrimination to be weak. However, the EuroSCORE overestimated mortality in patients at very high risk (O/E = 0.74, 95% CI 0.6 to 0.89, P =0.001; AUC = 0.69).

Table 3 Estimation of operative risk according to the risk group (additive EuroSCORE).

[ Table Omitted - see PDF ]

Estimation of operative risk according to the type of procedure

Table 4 shows the different relationship of the estimation of operative risk according to the type of surgical procedure performed. The EuroSCORE tended to overestimate mortality in isolated CABG operations (O/E = 0.81, 95% CI 0.63 to 1.05, P =0.0001; AUC = 0.71) and in isolated valve operations (O/E = 0.89, 95% CI 0.73 to 1.1, P =0.007; AUC=0.69). Moreover, the EuroSCORE tended to underestimate mortality in the combined operations (O/E = 1.09, 95% CI 0.98 to 1.23, P <0.0001; AUC=0.79).

Table 4 Operative risk estimation according to the type of procedure.

[ Table Omitted - see PDF ]

Estimation of operative risk according to the surgeon factor

The influence of the surgeon in estimating operative risk was variable, as shown in Table 5. The EuroSCORE overestimated mortality for the surgeon A (O/E = 0.46, 95% CI 0.35 to 0.61; P <0.0001, AUC = 0.89) and underestimated for the surgeon B (O/E = 1.3, 95% CI 1.09 to 1.56, P <0.0001; AUC=0.76). There was a tendency to overestimate mortality for surgeon C, and underestimate the surgeons D and E.

Table 5 Estimation of operative risk according to the surgeon factor.

[ Table Omitted - see PDF ]

Relationship between the surgeon factor and risk groups (additive EuroSCORE)

As shown in Table 5, there were differences in patient according to surgeons, surgeons C and A faced the most severe cases according to the EuroSCORE as surgeons D and E the less severe (Wilcoxon P =0.006). The surgeon A showed superior results when compared to the predicted by the EuroSCORE in all risk groups, unlike the surgeon B, as shown in Table 6. Results from other surgeons were variable.

Table 6 Relation between risk groups and the surgeon factor in estimating operative risk.

[ Table Omitted - see PDF ]

Relationship between the surgeon factor and the type of procedure

There were significant differences among surgeons with regard to the type of procedures performed by each. The surgeon C operated less isolated MRI and combined operations over all other surgeons who were similar in all sorts of procedures (P <0.0001).

The surgeon factor neutralized the biggest trend of risk in combined operations. In combined operations, surgeons A (O/E=0.72, 95% CI 0.57 to 0.94, P <0.0001; AUC=0.93) and C (O/E=0.8; 95% from 0.56 to 1.17, P =0.0008; AUC=0.87) showed better results than predicted by EuroSCORE, unlike surgeons B (O/E=1.24, 95% CI 0.98 - 1.56, P =0.0003; AUC = 0.79), D (O/E=1.74, 95% CI 1.45 to 2.15, P =0.01; AUC=0.69) and And (O/E=1.42, 95% CI 1.15 to 1.89, P =0.03; AUC=0.82), as shown in Table 7.

Table 7 Relation between types of procedure and the surgeon factor in estimating operative risk.

[ Table Omitted - see PDF ]

DISCUSSION

This study sought to examine the first version of the EuroSCORE, in order to validate the score in a Brazilian center for cardiovascular surgery, and determine the impact of the type of procedure and surgeon factor in predicting 30-day mortality.

We observed that the EuroSCORE correlates with overall mortality of our service as well as others in our country^[13] had already shown. The EuroSCORE overestimated mortality in patients at very high risk and there was still a tendency to overestimate mortality in low-risk patients. These findings are consistent with the most recent case series from developed countries^[14] , unlike other series^[6-8] have shown exactly the opposite.

The reasons for this discrepancy are various. First, there was a great progression of cases of perioperative care since the original publication of the EuroSCORE, which certainly had an impact in reducing operative mortality in centers of excellence. Moreover, these improvements were not incorporated in the same proportion and speed in centers of which the result is unsatisfactory. Structural problems in the health system and hospitals, teaming and unfavorable patient characteristics contribute to accentuate these differences. Thus, the risk scores present better performance when the preoperative characteristics and treatment regimens are comparable to those in which the score was derived.

Any risk score can only be used reliably when tested in the local population and when treatment regimens did not show substantial differences after the development of risk score^[9] . The recently launched review of the EuroSCORE II includes certain limitations of the first version, but needs validation in other populations. Only the validation of risk scores in different populations in different continents is essential for their clinical applicability. The creation of an own risk score that takes into account the peculiar characteristics of the population would be ideal ^[15] . Different risk scores may exist between the models and also with different weights that affect the risk prediction^[16] .

As regards the type of procedure, there was a tendency in this study to overestimate the mortality of isolated CABG and valve operations, and a tendency to underestimate the mortality of combined operations. Several studies in international^[17,18] literature have shown the same results in coronary operations. Although the original database from EuroSCORE presents a minority of valve operations, our findings are also consistent with other studies in the literature^[14,19,20] , although the characteristics of the patients in our country are particularly different for the prevalence of rheumatic disease, large proportion of reoperations and pulmonary hypertension.

The group of valve diseases has gained particular interest in the advent of percutaneous aortic prostheses, whose indication takes much into account the scores of operative risk as the current recommendation restricts inoperable^[21] or high surgical risk cases. However, the indication of these procedures must be decided with great care without relying solely on risk scores, as they proved to be inadequate for this purpose.

Regarding the combined operations, our findings are also consistent with other studies^[14] . This is the group most difficult to assess due to the variability of combining procedures. Due to the restricted denominator for each type individually, it is difficult to predict mortality accurately. The difficulty of comparison between hospitals and individuals is to adjust the complexity of cases among groups. Simple adjustment of mortality risk does not eliminate all selection biases and can influence the test results. However, the design of performance studies is observational and in the real world and with risk adjustment. Such analyzes would be comparable with a valid reference standard with the same characteristics of patients, usually within the same population of individuals ^[22] .

Considering the same limitations noted above, the impact of the surgeon factor may be important in influencing the severity of patients and the type of procedure. This was further supported by the negative effect exerted by the surgeon A on mortality by multivariate analysis. Generally, the surgeon had a positive impact on all grades of risk, particularly in severe cases and combined operations, unlike the surgeon B.

The influence of the surgeon unlike the hospital reinforces the notion that within a high complex service, heterogeneity of surgeons' performance may exist determining impact on results. The surgical volume, training and technical and clinical experience of each team member must be taken into consideration at the time of surgical scheduling, as well as the expertise and personal familiarity with each procedure to ensure the best possible outcome for the patient. The formation and maintenance of integrated multidisciplinary teams engaged in developing systematic health care programs and quality control ^[23] programs have shown positive impact on surgical outcomes in cardiac surgery^[24,25] .

Based on our findings, it is important that the risk score has constant updating and validation in Brazilian centers. The risk score should not be used alone in indicating whether or not a surgical procedure, without being part of a broader context of clinical surgical discussion, because there are unmeasured factors that may influence the risk prediction.

Limitations of the study

The present study has several limitations inherent to its design mainly. Because of the risk scores were constructed for use in large populations, analysis of subgroups with smaller denominator reduces the statistical power and can lead to misleading interpretations. The differences in the proportion of patients undergoing different degrees of complexity among surgeons may have caused selection bias. However, we were careful to adjust the mortality rates for the same severity group of patients in order to minimize these limitations. The ideal, however, would have been adjusted for each value of the risk score, which would entail further reduced denominator.

The authors acknowledge that there are no perfect mathematical models, and thus our findings may be subject to errors. Previous studies^[12,26] have shown that there can be large fluctuations results in the classification of hospitals and surgeons, whose final interpretation questioned the statistical validity of the model. However, there is validity to demonstrate these results as a form of quality control, as usually the critical analysis of the results leads to subsequent improvement in outcomes ^[27] .

CONCLUSION

In the population studied, the first version of the EuroSCORE overestimates mortality in patients at very high risk and may be influenced by the type of surgery proposed and the surgeon factor. We must also choose the most appropriate surgeon for each severity from patient to minimize the risk imposed by preoperative characteristics.

Authors' roles & responsibilities

FAA Analysis and interpretation of data; statistical analysis; final approval of the manuscript; conception and design of the study; writing of the manuscript

CRC Analysis and/or interpretation of data; final approval of the manuscript; conception and design of the study

No financial support.

This study was carried out at Institute of Cardiology of Distrito Federal (ICDF), Brasília, DF, Brazil.

References

1. Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16(1):9-13.

2. Michel P, Roques F, Nashef SA; EuroSCORE Project Group. Logistic or additive EuroSCORE for high-risk patients? Eur J Cardiothorac Surg. 2003;23(5):684-7.

3. Roques F, Nashef SA, Michel P, Pinna Pintor P, David M, Baudet E; EuroSCORE Study Group. Does EuroSCORE work in individual European countries? Eur J Cardiothorac Surg. 2000;18(1):27-30.

4. Nashef SA, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL, et al; EuroSCORE Project Group. Validation of European System for Cardiac Operative Risk Evaluation (EuroSCORE) in North American cardiac surgery. Eur J Cardiothorac Surg. 2002;22(1):101-5.

5. Yap CH, Reid C, Yii M, Rowland MA, Mohajeri M, Skillington PD, et al. Validation of the EuroSCORE model in Australia. Eur J Cardiothorac Surg. 2006;29(4):441-6.

6. Malik M, Chauhan F, Malik V, Gharde P, Kiran U, Pandey RM. Is EuroSCORE applicable to Indian patients undergoing cardiac surgery? Ann Card Anaesth. 2010;13(3):241-5.

7. Carvalho MRM, Souza e Silva NA, Klein CH, Oliveira GMM. Aplicação do EuroSCORE na cirurgia de revascularização miocárdica em hospitais públicos do Rio de Janeiro. Rev Bras Cir Cardiovasc. 2010;25(2):209-17.

8. Sá MPB, Soares EF, Santos CA, Figueiredo OJ, Lima ROA, Escobar RR, et al. EuroSCORE e mortalidade em cirurgia de revascularização miocárdica no Pronto Socorro Cardiológico de Pernambuco. Rev Bras Cir Cardiovasc. 2010;25(4):474-82.

9. Ivanov J, Tu JV, Naylor CD. Ready-made, recalibrated, or remodeled? Issues in the use of risk indexes for assessing mortality after coronary artery bypass graft surgery. Circulation. 1999;99(16):2098-104.

10. Karthik S, Srinivasan AK, Grayson AD, Jackson M, Sharpe DA, Keenan DJ, et al. Limitations of additive EuroSCORE for measuring risk stratified mortality in combined coronary and valve surgery. Eur J Cardiothorac Surg. 2004;26(2):318-22.

11. Asimakopoulos G, Al-Ruzzeh S, Ambler G, Omar RZ, Punjabi P, Amrani M, et al. An evaluation of existing risk stratification models as a tool for comparison of surgical performances for coronary artery bypass grafting between institutions. Eur J Cardiothorac Surg. 2003;23(6):935-41.

12. Siregar S, Groenwold RH, Jansen EK, Bots ML, van der Graaf Y, van Herwerden LA. Limitations of ranking lists based on cardiac surgery mortality rates. Cir Cardiovasc Qual Outcomes. 2012;5(3):403-9.

13. Mejia OAV, Lisboa LAF, Puig LB, Dias RR, Dallan LA, Pomerantzeff PM, Stolf NAG. Os escores 2000 Bernstein-Parsonnet e EuroSCORE são similares na predição da mortalidade no Instituto do Coração-USP. Rev Bras Cir Cardiovasc. 2011;26(1):1-6.

14. Ranucci M, Castelvecchio S, Menicanti LA, Scolletta S, Biagioli B, Giomarelli P. An adjusted EuroSCORE model for high-risk cardiac patients. Eur J Cardiothorac Surg. 2009;36(5):791-7.

15. Braile DM, Monteiro R, Brandau R, Jatene FB. Modelos de predição de risco: são eles realmente necessários? Arq Bras Cardiol. 2010;95(6):677-8.

16. Grant SW, Grayson AD, Jackson M, Au J, Fabri BM, Grotte G, et al. Does the choice of risk-adjustment model influence the outcome of surgeon-specific mortality analysis? A retrospective analysis of 14,637 patients under 31 surgeons. Heart. 2008;94(8):1044-9.

17. Parolari A, Pesce LL, Trezzi M, Loardi C, Kassem S, Brambillasca C, et al. Performance of EuroSCORE in CABG and off-pump coronary artery bypass grafting: single institution experience and meta-analysis. Eur Heart J. 2009;30(3):297-304.

18. Zheng Z, Li Y, Zhang S, Hu S; Chinese CABG Registry Study. The Chinese coronary bypass grafting registry study: how well does the EuroSCORE predict operative risk for Chinese population? Eur J Cardiothorac Surg. 2009;35(1):54-8.

19. Parolari A, Pesce LL, Trezzi M, Cavallotti L, Kassem S, Loardi C, et al. EuroSCORE performance in valve surgery: a meta-analysis. Ann Thorac Surg. 2010,89(3):787-93.

20. Andrade ING, Moraes Neto FR, Oliveira JPSP, Silva ITC, Andrade TG, Moraes CRR. Avaliação do EuroSCORE como preditor de mortalidade em cirurgia cardíaca valvar no Instituto do Coração de Pernambuco. Rev Bras Cir Cardiovasc. 2010;25(1):11-8.

21. Leon MB, Smith CR, Mack M, Miller DC, Moses JW, Svensson LG, et al; PARTNER Trial Investigators. Transcatheter aortic-valve implantation for aortic stenosis in patients who cannot undergo surgery. N Engl J Med. 2010;363(17):1597-607.

22. Shahian DM, Normand SL. Comparison of "risk-adjusted" hospital outcomes. Circulation. 2008;117(15):1955-63.

23. Kurlansky PA, Argenziano M, Dunton R, Lancey R, Nast E, Stewart A, et al. Quality, not volume, determines outcome of coronary artery bypass surgery in a university-based community hospital network. J Thorac Cardiovasc Surg. 2012;143(2):287-93.

24. Atik FA, Garcia MFMA, Santos LM, Chaves RB, Faber CN, Corso RB, et al. Resultados da implementação de modelo organizacional de um serviço de cirurgia cardiovascular. Rev Bras Cir Cardiovasc 2009;24:116-25.

25. Atik FA. Quality improvement program decreases mortality after cardiac surgery. J Thorac Cardiovasc Surg. 2009;138(1):253-4.

26. Green J, Wintfeld N. Report cards on cardiac surgeons. Assessing New York State's approach. N Engl J Med. 1995;332(18):1229-32.

27. Hannan EL, Sarrazin MS, Doran DR, Rosenthal GE. Provider profiling and quality improvement efforts in coronary artery bypass graft surgery: the effect on short-term mortality among Medicare beneficiaries. Med Care. 2003;41(10):1164-72.