Predictive ability of scores for bleeding risk in heart disease outpatients on warfarin in Brazil

Abstract

Introduction

Bleeding is a common complication in patients taking warfarin. We sought to compare the performance of nine prediction models for bleeding risk in warfarin-treated Brazilian outpatients.

Methods

The dataset was derived from a clinical trial conducted to evaluate the efficacy of an anticoagulation clinic at a public hospital in Brazil. Overall, 280 heart disease outpatients taking warfarin were enrolled. The prediction models OBRI, Kuijer et al., Kearon et al., HEMORR2HAGES, Shireman et al., RIETE, HAS-BLED, ATRIA and ORBIT were compared to evaluate the overall model performance by Nagelkerke’s R2 estimation, discriminative ability based on the concordance (c) statistic and calibration based on the Hosmer-Lemeshow goodness-of-fit statistic. The primary outcomes were the first episodes of major bleeding, clinically relevant non-major bleeding and non-major bleeding events within 12 months of follow-up.

Results

Major bleeding occurred in 14 participants (5.0%), clinically relevant non-major bleeding in 29 (10.4%), non-major bleeding in 154 (55.0%) and no bleeding at all in 115 (41.1%). Most participants with major bleeding had their risk misclassified. All the models showed low overall performance (R2 0.6–9.3%) and poor discriminative ability for predicting major bleeding (c <0.7), except Shireman et al. and ORBIT models (c 0.725 and 0.719, respectively). Results were not better for predicting other bleedings. All models showed good calibration for major bleeding.

Conclusions

Only two models (Shireman et al. and ORBIT) showed at least acceptable performance in the prediction of major bleeding in warfarin-treated Brazilian patients. Accurate models warrant further investigation to be used in similar populations.

About the Authors:

João Antonio de Queiroz Oliveira

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Software, Writing – original draft, Writing – review & editing

Affiliations Faculdade de Medicina da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

ORCID logo http://orcid.org/0000-0003-3116-4713

Antonio Luiz Pinho Ribeiro

Roles Conceptualization, Funding acquisition, Project administration, Resources, Software, Supervision, Visualization, Writing – review & editing

Affiliations Faculdade de Medicina da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Daniel Dias Ribeiro

Roles Visualization, Writing – review & editing

Affiliation: Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Vandack Nobre

Roles Conceptualization, Supervision, Visualization, Writing – review & editing

Affiliations Faculdade de Medicina da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Manoel Otávio da Costa Rocha

Roles Conceptualization, Visualization, Writing – review & editing

Affiliations Faculdade de Medicina da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Maria Auxiliadora Parreiras Martins

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Faculdade de Medicina da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Hospital das Clínicas da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil, Faculdade de Farmácia da Universidade Federal de Minas Gerais, Belo Horizonte, Minas Gerais, Brazil

Introduction

Anticoagulation therapy has been proved to be effective in reducing the risk of thromboembolic events. Although vitamin K antagonists, such as warfarin, are widely used for the prevention of these events, bleeding complications are not uncommon [1]. Annual incidence of warfarin-related bleeding varies considerably according to study design and indication, intensity and duration of anticoagulation therapy, ranging from 1.1 to 3.4% for major bleeding [2].

Several methods to estimate the risk for warfarin-related bleeding have been developed, adapted and validated [3–14]. Accurate models for prediction of bleeding risk are useful for medical practices and patient care during oral anticoagulation treatment reducing the risk of life-threatening bleeding complications and potentially improving the efficacy of drug therapy by providing individualized care to patients with higher bleeding risk. However, there is a lack of information about the performance of these models in populations from low and middle-income countries, like Brazil. The aim of this study was to compare the performance of the main models for predicting bleeding risk found in literature in a Brazilian group of outpatients with heart diseases on warfarin.

Methods

Study design, setting and participants

A prospective study was performed to compare the predictive ability of nine scores used to evaluate the bleeding risk in patients using warfarin as anticoagulation therapy. The data analyzed in the present study was derived from a clinical trial conducted to evaluate the efficacy of an anticoagulation clinic at a university public hospital located in Southeast Brazil. The main condition for the admission of patients at this clinic is the baseline diagnosis of cardiopathy, such as atrial fibrillation, rheumatic valve disease and Chagas cardiomyopathy. This trial included patients aged 18 years or older presenting a diagnosis of cardiac disease and with at least one indication of long-term oral anticoagulation, such as atrial fibrillation/flutter, mechanical heart valves, history of ischemic stroke/transient ischemic attack or thrombosis. During a one-year follow up, information regarding bleeding episodes was collected from 280 patients included in the trial. Then, we analyzed the performance of nine predictive models for warfarin-related bleeding risk in this population. The trial was approved by the Research Ethics Committee of the Federal University of Minas Gerais (ETIC 376/09) and it was registered at ClinicalTrials.gov (NCT01006486). All study participants signed an informed consent form.

Variables and classification of bleeding risk

Demographic, clinical and laboratory data were used to calculate the bleeding risk according to the OBRI [5], Kuijer et al. [6], Kearon et al. [15], HEMORR2HAGES [7], Shireman et al. [8], RIETE [9], HAS-BLED [10], ATRIA [12] and ORBIT [16] prediction models (Table 1).

[Figure omitted. See PDF.]

Table 1. Risk factors and categories proposed by the prediction models for bleeding.

https://doi.org/10.1371/journal.pone.0205970.t001

Genetic data required for calculation of bleeding risk by the HEMORR2HAGES [7] model was not available. In the risk stratification of the prediction model proposed by the Shireman et al. [8], any previous bleeding events were considered as remote bleeding due to the absence of information about the time of the prior bleeding occurrence. Data concerning previous peptic ulcer disease and drug abuse were not available. For RIETE score [9], recent major bleeding was defined as any prior major bleeding. For all the cases in which data was not available, the risk factors were assigned as absent.

The time in therapeutic range (TTR) was determined by the Rosendaal et al. method [17]. We used the follow-up international normalized ratio (INR) measurements to determine the TTR, considering the INRs from baseline till the first major bleeding event or all the INRs during the follow-up for patients with no major bleeding. In ATRIA [12] and ORBIT [16] we assigned severe renal disease and insufficient kidney function, respectively, as risk factors based on the estimated glomerular filtration rate (eGFR) using the creatinine clearance calculated by the Cockcroft-Gault formula [18].

Outcomes

The primary outcome variables were the first episodes of major bleeding and clinically relevant non-major bleeding events within one year of follow-up. Although the models have not been originally developed for predicting clinically relevant non-major bleeding and non-major bleeding, this outcome was included as an exploratory endpoint, similar to previous comparative studies found in literature [13,19,20]. The severity of bleeding outcomes was classified according to the criteria recommended from the Subcommittee on Control of Anticoagulation of the Scientific and Standardization Committee of the International Society on Thrombosis and Haemostasis [21]. Major bleeding was defined as a fatal bleeding; and/or bleeding in a critical area or organ (e. g. intracranial, retroperitoneal); and/or bleeding causing fall ≥2 g/dL in hemoglobin level, or requiring transfusion of ≥2 units of red blood cells. Fatal bleeding was defined as a bleeding that directly caused death without any other identifiable cause. Clinically relevant non-major bleeding was defined as any overt sign or symptom of bleeding (e.g., more bleeding than would be expected for a clinical circumstance, including bleeding found by imaging alone) that does not fit the criteria for major bleeding but does meet at least one of the following criteria: requiring medical intervention by a healthcare professional; leading to hospitalization or increased level of care; and/or unscheduled face to face contact with a physician for evaluation. Non-major bleeding was defined as events that do not meet the criteria for major bleeding or clinically relevant non-major bleeding [21].

Statistical analysis

We used descriptive methods to evaluate baseline characteristics. Continuous variables were tested for normality by Kolmogorov-Smirnov test. Continuous variables are presented as mean ± standard deviation (SD) or median (interquartile range), as appropriate, and categorical variables as percentage. Subgroups were compared by Student t-test or Mann-Whitney U tests, when indicated.

As a measurement of the overall model performance, the Nagelkerke’s R2 was estimated to provide the magnitude that the model can predict the outcome of interest. The Nagelkerke’s R2 may vary from 0.00 (0.0%) to 1.00 (100%) and a higher number indicates a better predictability of the model. Discriminative ability was measured by the concordance (c) statistic, which is equal to the area under the curve (AUC) of a receiver operating characteristic (ROC) for a binary outcome, and varies between no discrimination (0.50), poor (0.51–0.69), acceptable (0.70–0.79), excellent (0.80–0.89), outstanding (0.90–0.99) and perfect discrimination (1.00) [22]. AUC values lower than 0.5 indicates that the model performance is worse than a random assumption. We assessed the calibration of the models by the Hosmer-Lemeshow goodness-of-fit (HL-GOF) statistic. Non-significant HL-GOF statistics indicate a well-fitting model. Observed versus expected probabilities plots were also used to assess calibration for major bleeding outcomes. The models were compared using two different ways: a) quantitative models comparison, using the total risk assessed; and b) qualitative models comparison, using the classification of risk (i.e., low-intermediate and high bleeding risk). Statistical significance was considered when p <0.05. Sensitivity analysis was done assigning as present the factors in which we previously had assigned as absent due to insufficient data to evaluate, i.e. peptic ulcer disease (1 point) in Kearon et al. [15] model, genetic factor (1 point) in HEMORR2HAGES [7] and drug abuse (0.71 point) in Shireman et al. [8] model. We also assigned for patients who had history of bleeding, previously assigned as remote bleeding (0.58 point) due to the lack of occurrence time information, as recent bleeding (0.62 point) in Shireman et al. [8] model. All statistical analyses were performed using IBM SPSS Statistics for Windows, Version 21.0 (IBM Corp, Armonk, NY).

Results

Study participants’ characteristics and outcomes

The demographic and clinical characteristics of the participants stratified by the severity of the bleeding events are summarized in Table 2. Participants had a mean age of 56.8±13.1 years and they were predominantly female (54.6%). Most common clinical characteristics were atrial fibrillation/flutter (63.3%), hypertension (60.0%), history of non-major bleeding (65.7%) and excessive fall risk (52.9%). Almost a third of participants were diagnosed with Chagas disease. At least one episode of major bleeding occurred in 14 participants (5.0%), clinically relevant non-major bleeding in 29 (10.4%), non-major bleeding in 154 (55.0%) and no bleeding at all in 115 (41.1%). Atrial fibrillation/flutter (85.7%), mechanical heart valve (64.3%), hypertension (64.3%) and previous non-major bleeding (92.9%) were more frequent among the participants who had major bleeding during the follow-up.

[Figure omitted. See PDF.]

Table 2. Patients’ characteristics including risk factors proposed by the prediction models with stratification by severity and the absence or presence of the bleeding events.

https://doi.org/10.1371/journal.pone.0205970.t002

Classification of bleeding risk

Most participants were classified in the intermediate risk category using OBRI [5] (56.1%), Kuijer et al. [6] (72.1%) and HAS-BLED [10] (49.6%) while Kearon et al. [15] and HEMORR2HAGES [7] prediction models classified the participants mainly in the high risk category (51.4% and 47.5%, respectively). ATRIA [12] model classified the majority of the participants who had bleeding, independently of its severity, and those who did not have bleeding in the low risk category. Among the participants who had major bleeding during the follow-up, the majority were classified in the high risk category by Kearon et al. [15], HEMORR2HAGES [7], HAS-BLED [10] and ORBIT [16], as well as those participants who had clinically relevant non-major bleeding, except HAS-BLED [10] and ORBIT [16] models which classified the majority of these participants in the low-intermediate risk categories. None participant who had major bleeding and/or clinically relevant non-major bleeding was classified in the high risk category using Kuijer et al. [6] and RIETE [9] models (Table 3).

[Figure omitted. See PDF.]

Table 3. Classification of bleeding risk prediction for the total number of participants and its stratification by severity and by the absence or presence of bleeding events.

https://doi.org/10.1371/journal.pone.0205970.t003

Comparison of the prediction models

For quantitative models comparison, the discriminative ability of the models for predicting major bleeding, expressed as c statistics, ranged from 0.546 to 0.725. Only Shireman et al. and ORBIT models achieved an acceptable level of predictive performance while the others showed poor discriminative ability (c index <0.70). All the models showed poor predictive performance when considering clinically relevant non-major bleeding and non-major bleeding as the main outcome, for which the discriminative ability ranged from 0.407 to 0.559, and from 0.438 to 0.582, respectively. There was a significant difference between the discriminative ability of the models for predicting major bleeding and non-major bleeding (p<0.05) (Table 4, Fig 1). The results of discriminative ability in qualitative models comparison were similar to the quantitative approach (S1 Table).

[Figure omitted. See PDF.]

Table 4. Discriminative ability of the prediction models (quantitative approach).

https://doi.org/10.1371/journal.pone.0205970.t004

[Figure omitted. See PDF.]

Fig 1. Receiver operating characteristic curves of the models for predicting bleeding risk of major, clinically relevant non-major and non-major bleeding outcomes.

https://doi.org/10.1371/journal.pone.0205970.g001

All models presented low Nagelkerke’s R2 values, which ranged from 0.6% to 9.3% for predicting major bleeding, from 0.0% to 2.0% for clinically relevant non-major bleeding and from 0.0% to 2.7% for predicting non-major bleeding, indicating a poor overall performance. All the models presented no significant difference between their predictive ability and the observed major bleedings (Table 5). Calibration plots showed high concordance between the risk predicted by all models and the observed major bleeding events (S1 Fig). Only Kearon et al. [15] did not fit well for predicting clinically relevant non-major bleeding (p = 0.016) and Kuijer et al. [6] and ATRIA [12] did not fit well for predicting non-major bleeding (p = 0.001 and p = 0.004, respectively) (Table 5). The results of overall performance were similar when the models were compared using the qualitative approach (S2 Table).

[Figure omitted. See PDF.]

Table 5. Overall performance and calibration of the prediction models (quantitative approach).

https://doi.org/10.1371/journal.pone.0205970.t005

After sensitivity analysis, no models showed significant enhance of the discriminatory ability or significant improvement in the overall performance. Detailed results of sensitivity analysis can be found in Supporting Information S3, S4 and S5 Tables.

Discussion

We found that only two (Shireman et al. and ORBIT) of the models tested achieved an acceptable level of discriminative ability for predicting major bleeding while the others showed poor prediction performance. Among the patients who had experienced major bleeding, only the Kearon et al. [15] and HAS-BLED [10] schemes classified the majority in the high-risk category. All the models showed a poor discriminative ability, as reflected by a c statistic value lower than 0.7 for predicting clinically relevant non-major and non-major bleeding. Similarly, the overall performance was not good considering the low R2 values for all outcomes. All models fit well with the data for predicting major bleeding when the models were compared using quantitative approach.

The majority of the participants who experienced major bleeding were also misclassified by the models in other studies [11,13,14,19]. In the comparative validation study performed by Lip et al. [11], the majority of 136 patients who had major bleeding were stratified in moderate (intermediate) risk category by HAS-BLED, OBRI and Kuijer et al. models and in low risk category by HEMORR2HAGES and Shireman et al. models, with none of these patients classified in high risk category by the last one. Similar results were found by Donzé et al. [14] for OBRI, Kuijer et al., Shireman et al. and HAS-BLED. This study also included the RIETE model, which classified the majority of patients into the intermediate category, and ATRIA model, which classified almost the same proportion into the low and the high categories. The high variability in the classification schemes for bleeding risk, especially in participants who had major bleeding, could be explained by differences regarding the clinical characteristics of the participants. For instance, those participants who had mechanical heart valve which requires, in general, a more intense anticoagulation, have an increased risk of bleeding. No models took this fact into account as a risk factor, although the OBRI [5] model included these patients in the validation sample. This could also explain why the majority of the participants who had major bleeding in our study had this condition.

The c statistic reported on models’ internal validation studies [5–8,10,12], considering only vitamin K antagonists-treated participants, ranged from 0.632 to 0.82 showing higher results. The discriminative ability achieved an acceptable level only in OBRI [5] (c statistic 0.72) and an excellent level in Kuijer et al. [6] (0.82) derivation study. The c statistic was not available in the Kearon et al. [15] and the RIETE [9] studies. A more restrict inclusion criteria is one possible reason for the better discriminative ability showed by the models in their original study, as some of them included only patients with venous thromboembolism. This selected group often does not present multiple comorbidities as heart disease patients commonly do. Thus, there would be limitations for its use in populations with other indications of anticoagulation therapy. It is also noteworthy that, in general, all the models had better performance to predict major bleedings when compared to the others levels of severity. This finding was expected since all studied models were developed and validated to predict major bleedings.

Our results reinforce previous results of other comparative studies showing poor overall performance of predictive models [11,13,14,23]. In a recent study with unselected patients receiving oral anticoagulation therapy, Donzé et al. [14] compared seven bleeding risk prediction models and found c statistic values ranged from 0.54 to 0.61, concluding that there were no difference between the models and physicians prediction for bleeding risk. In another study, Apostolakis et al. [13] demonstrated only modest performance of HEMORR2HAGES, HAS-BLED and ATRIA models in atrial fibrillation patients, with a slightly better performance of the HAS-BLED model for predicting major and any clinical relevant bleeding. However, the c statistic ranged from 0.61 to 0.65 for major bleeding and from 0.50 to 0.60 for any clinically relevant bleeding outcome. In a comparative validation, Lip et al. [11] demonstrated a slightly higher c statistic value of HAS-BLED (0.66) than four other models (ranged 0.52 to 0.63). Roldán et al. [23], in a two-model comparison study, found a significant better discriminative ability of HAS-BLED over ATRIA when they were analyzed as dichotomized models (0.68 vs 0.59, p = 0.035). Even though some models performed slightly better than others, these studies found c statistic values lower than 0.7, indicating poor discriminative ability levels, which were similar to our results for the majority of the models tested, suggesting that these models may not be useful for predicting major, clinically relevant non-major or non-major bleeding. Remarkably, the highest and the only acceptable performance in our study was showed by Shireman et al. and ORBIT models, the latter being the most recent model and one of the simplest to apply in clinical practice.

The discrepant results between the studies of model internal validation and our results could be explained by differences regarding study design and characteristics of the included populations. Some models were developed in studies with short follow-up, such as the studies performed by Kuijer et al. [6], Shireman et al. [8] and RIETE [9] which considered a follow-up period of three months, in contrast with the long-term follow-up performed by OBRI (4 years) [5], Kearon et al. (2.4 years in average) [15], HEMORR2HAGES (3 years) [7], HAS-BLED (1 year) [10], ATRIA (6 years) [12] and ORBIT (2 years in average) [16]. The incidence of bleeding events tends to increase as the period of observation becomes longer. This variation could impact on the detection of risk factors for major bleeding and their weight in the development process of the models. The characteristics of the type of patient care may have affected the determination of bleeding risk. OBRI [5], Kuijer et al. [6], RIETE [9] and ORBIT [16] models derived from data of outpatients while HEMORR2HAGES [7], Shireman et al. [8] and ATRIA [12] were developed using data of hospitalized patients. On the other hand, HAS-BLED [10] derived from data of ambulatory and hospitalized patients. Most models used database from previous studies designed for other purposes resulting in limitations of the identification of risk factors for bleeding. Regarding our population, Chagas disease is an important and common cause of cardiomyopathy in Brazil, which increases the risk of thromboembolism events. Although this disease has not been considered as a risk factor for bleeding by the predictive models studied, the considerable frequency of chagasic patients presenting major bleeding deserves to be further investigated.

Limitations and strengths

Some limitations of the present study should be addressed. Firstly, there is a possibility of bias due to the use of a clinical trial database, which was designed for another purpose. There was no data to determine the presence or absence of some risk factors affecting the categorization of bleeding risk. The lack of genetic information could have influenced on HEMORR2HAGES [7] classification of risk, although their derivation study did not use this information either. In addition, after performing sensitivity analyses the findings showed to be consistent with those from the primary analysis and lead to similar conclusions. Secondly, as participants with different indications for oral anticoagulation were included, there was no evaluation of the models performance by each indication separately, and many of them differ from the model originally developed. Thirdly, the performance of the models was assessed for predicting not only major, but also clinically relevant non-major and non-major bleeding events, even though they have been developed for predicting major bleeding events only. However, all the models showed poor performance for all outcomes, which demonstrates that they are not good at predicting bleeding events independently of the severity. Finally, we conducted our study at a single university hospital, and the generalizability of our results to other anticoagulation clinics across Brazil and South America remains unclear.

Despite these limitations, this was the first study comparing prediction models of bleeding risk in a population with a low socioeconomic status of a middle-income country and also including a significant proportion of patients presenting Chagas disease or rheumatic valvulopathy. The health systems in low-income and middle-income countries suffer from scarce financial and human resources when compared to high-income countries. Besides, populations from these countries may experience worse conditions of education, health literacy, income and access to high quality health services than high-income countries. Accurate prediction models for bleeding could contribute to the identification of outpatients at higher risks for warfarin complications and to the individualization of the provision of patient care. Consequently, hospital admissions due to bleeding events could be reduced, as well as the costs related to medical interventions. In Brazil, some of the prediction models have been increasingly used to assess the risk of bleeding in patients starting oral anticoagulation. However, our findings suggest that the use of the models tested in clinical practice may not be as useful as expected and highlights the potential need of developing a prediction model derived from similar populations in South America. We recently demonstrated that a dedicated anticoagulation clinic improves the quality and safety of anticoagulation therapy in Brazilian patients [24]. Accurate models to predict bleeding complication could be used to enhance even more the quality of care.

Conclusions

The comparison of the performance of nine scores for bleeding risk in heart disease outpatients using warfarin in Brazil demonstrated that only Shireman et al. and ORBIT models showed acceptable ability to predict major bleeding, and none showed good performance to predict clinically relevant non-major and non-major bleeding events. The tested models have significant limitations to assess the bleeding risk in warfarin-treated patients and to guide clinical decision on the management of anticoagulation therapy. More accurate models should be investigated to improve the stratification of the bleeding risk and, consequently, to help provide an individualized care to patients, especially in low and middle-income countries from Brazil and other South American countries.

Supporting information

[Figure omitted. See PDF.]

S1 Table. Discriminative ability of the prediction models (dichotomized models).

https://doi.org/10.1371/journal.pone.0205970.s001

(PDF)

S2 Table. Overall performance and calibration of the prediction models (dichotomized models).

https://doi.org/10.1371/journal.pone.0205970.s002

(PDF)

S3 Table. Classification for predicting bleeding risk of the total participants and its stratification by severity and the absence or presence of the bleeding events (sensitivity analysis).

https://doi.org/10.1371/journal.pone.0205970.s003

(PDF)

S4 Table. Discriminatory ability for the prediction models (sensitivity analysis).

https://doi.org/10.1371/journal.pone.0205970.s004

(PDF)

S5 Table. Overall performance and calibration of the prediction models (sensitivity analysis).

https://doi.org/10.1371/journal.pone.0205970.s005

(PDF)

S1 Fig. Plots of observed proportion of major bleeding events in the data versus expected proportion from each model.

https://doi.org/10.1371/journal.pone.0205970.s006

(PDF)

S1 Dataset. All relevant data available.

https://doi.org/10.1371/journal.pone.0205970.s007

(XLSX)

Acknowledgments

All authors satisfy the four criteria for authorship recommended by International Committee of Medical Journal Editors (ICMJE), as specified below: substantial contributions to the conception or design of the work (JAQO, ALPR, MAPM) and the acquisition, analysis, or interpretation of data for the work (all authors); drafting the work or revising it critically for important intellectual content (all authors); final approval of the version to be published (all authors); agreement to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved (all authors).

Citation: Oliveira JAdQ, Pinho Ribeiro AL, Ribeiro DD, Nobre V, Rocha MOdC, Martins MAP (2018) Predictive ability of scores for bleeding risk in heart disease outpatients on warfarin in Brazil. PLoS ONE 13(10): e0205970. https://doi.org/10.1371/journal.pone.0205970

References

1. Ageno W, Gallus AS, Wittkowsky A, Crowther M, Hylek EM, Palareti G (2012) Oral anticoagulant therapy: Antithrombotic Therapy and Prevention of Thrombosis, 9th ed: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines. Chest 141: e44S–88S.

2. Schulman S, Beyth RJ, Kearon C, Levine MN (2008) Hemorrhagic complications of anticoagulant and thrombolytic treatment: American College of Chest Physicians Evidence-Based Clinical Practice Guidelines (8th Edition). Chest 133: 257S–298S. pmid:18574268

3. Landefeld CS, Cook EF, Flatley M, Weisberg M, Goldman L (1987) Identification and preliminary validation of predictors of major bleeding in hospitalized patients starting anticoagulant therapy. Am J Med 82: 703–713. pmid:3494397

4. Landefeld CS, Goldman L (1989) Major bleeding in outpatients treated with warfarin: incidence and prediction by factors known at the start of outpatient therapy. Am J Med 87: 144–152. pmid:2787958

5. Beyth RJ, Quinn LM, Landefeld CS (1998) Prospective evaluation of an index for predicting the risk of major bleeding in outpatients treated with warfarin. Am J Med 105: 91–99. pmid:9727814

6. Kuijer PM, Hutten BA, Prins MH, Buller HR (1999) Prediction of the risk of bleeding during anticoagulant treatment for venous thromboembolism. Arch Intern Med 159: 457–460. pmid:10074953

7. Gage BF, Yan Y, Milligan PE, Waterman AD, Culverhouse R, Rich MW, et al. (2006) Clinical classification schemes for predicting hemorrhage: results from the National Registry of Atrial Fibrillation (NRAF). Am Heart J 151: 713–719. pmid:16504638

8. Shireman TI, Mahnken JD, Howard PA, Kresowik TF, Hou Q, Ellerbeck EF (2006) Development of a contemporary bleeding risk model for elderly warfarin recipients. Chest 130: 1390–1396. pmid:17099015

9. Ruiz-Gimenez N, Suarez C, Gonzalez R, Nieto JA, Todoli JA, Samperiz AL, et al. (2008) Predictive variables for major bleeding events in patients presenting with documented acute venous thromboembolism. Findings from the RIETE Registry. Thromb Haemost 100: 26–31. pmid:18612534

10. Pisters R, Lane DA, Nieuwlaat R, de Vos CB, Crijns HJ, Lip GY (2010) A novel user-friendly score (HAS-BLED) to assess 1-year risk of major bleeding in patients with atrial fibrillation: the Euro Heart Survey. Chest 138: 1093–1100. pmid:20299623

11. Lip GY, Frison L, Halperin JL, Lane DA (2011) Comparative validation of a novel risk score for predicting bleeding risk in anticoagulated patients with atrial fibrillation: the HAS-BLED (Hypertension, Abnormal Renal/Liver Function, Stroke, Bleeding History or Predisposition, Labile INR, Elderly, Drugs/Alcohol Concomitantly) score. J Am Coll Cardiol 57: 173–180. pmid:21111555

12. Fang MC, Go AS, Chang Y, Borowsky LH, Pomernacki NK, Udaltsova N, et al. (2011) A new risk scheme to predict warfarin-associated hemorrhage: The ATRIA (Anticoagulation and Risk Factors in Atrial Fibrillation) Study. J Am Coll Cardiol 58: 395–401. pmid:21757117

13. Apostolakis S, Lane DA, Guo Y, Buller H, Lip GY (2012) Performance of the HEMORR(2)HAGES, ATRIA, and HAS-BLED bleeding risk-prediction scores in patients with atrial fibrillation undergoing anticoagulation: the AMADEUS (evaluating the use of SR34006 compared to warfarin or acenocoumarol in patients with atrial fibrillation) study. J Am Coll Cardiol 60: 861–867. pmid:22858389

14. Donze J, Rodondi N, Waeber G, Monney P, Cornuz J, Aujesky D (2012) Scores to predict major bleeding risk during oral anticoagulation therapy: a prospective validation study. Am J Med 125: 1095–1102. pmid:22939362

15. Kearon C, Ginsberg JS, Kovacs MJ, Anderson DR, Wells P, Julian JA, et al. (2003) Comparison of low-intensity warfarin therapy with conventional-intensity warfarin therapy for long-term prevention of recurrent venous thromboembolism. N Engl J Med 349: 631–639. pmid:12917299

16. O’Brien EC, Simon DN, Thomas LE, Hylek EM, Gersh BJ, Ansell JE, et al. (2015) The ORBIT bleeding score: a simple bedside score to assess bleeding risk in atrial fibrillation. Eur Heart J 36: 3258–3264. pmid:26424865

17. Rosendaal FR, Cannegieter SC, van der Meer FJ, Briet E (1993) A method to determine the optimal intensity of oral anticoagulant therapy. Thromb Haemost 69: 236–239. pmid:8470047

18. Cockcroft DW, Gault MH (1976) Prediction of creatinine clearance from serum creatinine. Nephron 16: 31–41. pmid:1244564

19. Burgess S, Crown N, Louzada ML, Dresser G, Kim RB, Lazo-Langner A (2013) Clinical performance of bleeding risk scores for predicting major and clinically relevant non-major bleeding events in patients receiving warfarin. J Thromb Haemost 11: 1647–1654. pmid:23848301

20. Wells PS, Forgie MA, Simms M, Greene A, Touchie D, Lewis G, et al. (2003) The outpatient bleeding risk index: validation of a tool for predicting bleeding rates in patients treated for deep venous thrombosis and pulmonary embolism. Arch Intern Med 163: 917–920. pmid:12719200

21. Kaatz S, Ahmad D, Spyropoulos AC, Schulman S (2015) Definition of clinically relevant non-major bleeding in studies of anticoagulants in atrial fibrillation and venous thromboembolic disease in non-surgical patients: communication from the SSC of the ISTH. J Thromb Haemost 13: 2119–2126. pmid:26764429

22. Hosmer DW, Lemeshow S, Sturdivant RX (2013) Applied Logistic Regression. 3 ed. Hoboken, New Jersey: John Wiley & Sons.

23. Roldan V, Marin F, Fernandez H, Manzano-Fernandez S, Gallego P, Valdes M, et al. (2013) Predictive value of the HAS-BLED and ATRIA bleeding scores for the risk of serious bleeding in a "real-world" population with atrial fibrillation receiving anticoagulant therapy. Chest 143: 179–184. pmid:22722228

24. Martins MA, Oliveira JA, Ribeiro DD, César CC, Nobre VA, Palhares DM, et al. (2014) Abstract 13345: Efficacy and Safety of an Anticoagulation Clinic in Low-income Brazilian Patients With Heart Disease: a Randomized Clinical Trial. Circulation 130: A13345.