Measuring Appropriate Antibiotic Prescribing in Acute Hospitals: Development of a National Audit Tool Through a Delphi Consensus

Abstract

This study developed a patient-level audit tool to assess the appropriateness of antibiotic prescribing in acute National Health Service (NHS) hospitals in the UK. A modified Delphi process was used to evaluate variables identified from published literature that could be used to support an assessment of appropriateness of antibiotic use. At a national workshop, 22 infection experts reached a consensus to define appropriate prescribing and agree upon an initial draft audit tool. Following this, a national multidisciplinary panel of 19 infection experts, of whom only one was part of the workshop, was convened to evaluate and validate variables using questionnaires to confirm the relevance of each variable in assessing appropriate prescribing. The initial evidence synthesis of published literature identified 25 variables that could be used to support an assessment of appropriateness of antibiotic use. All the panel members reviewed the variables for the first round of the Delphi; the panel accepted 23 out of 25 variables. Following review by the project team, one of the two rejected variables was rephrased, and the second neutral variable was re-scored. The panel accepted both these variables in round two with a 68% response rate. Accepted variables were used to develop an audit tool to determine the extent of appropriateness of antibiotic prescribing at the individual patient level in acute NHS hospitals through infection expert consensus based on the results of a Delphi process.

1. Introduction

Antimicrobial resistance (AMR) is a global threat to health affecting all healthcare systems and growing at an alarming pace [1,2]. In the UK an increase in annual secondary care antibiotic consumption by 4.8% (measured as defined daily doses (DDD) of antibiotics per 1000 inhabitants per day) was reported in England between 2013 to 2017 [3]. Optimising prescribing, through the development and implementation of antimicrobial stewardship (AMS) programmes is a key area of the UK five-year AMR strategy [4]. An important element of these programmes is assessing the quality of antimicrobial prescribing. Several countries now use European Centre for Disease Prevention and Control (ECDC) point prevalence survey (PPS) of healthcare-associated infections and antimicrobial use protocol to identify targets for quality improvement and evaluate antimicrobial stewardship programmes [5]. Surveillance data collected via PPS provide useful information for assessment of the burden of antimicrobial prescribing, however, the method has limited capacity to assess the appropriateness of prescribing in the absence of more complete individual patient-level clinical data [6]. An adaptation of the ECDC point prevalence survey tool was developed by infection experts in Australia to better measure prescribing quality; this tool performed well in validation, inter-rater reliability testing, and user feedback [7]. The Centers for Disease Control and Prevention (CDC) in the US have also published an audit tool to evaluate the quality of inpatient antibiotic prescribing with similar positive feedback [8]. These tools share common themes in their evaluation of appropriateness, such as use of individual patient clinical data and indication to aid the auditor’s judgement of appropriateness. Two important limitations of existing audit protocols remain evident: Firstly, application of a broad definition of inappropriateness that includes a range of prescribing quality indicators (such as prompt intravenous-to-oral switch and appropriate dosing) that do not predict overall consumption of antibiotics; and secondly, an inability to specifically quantify the scale of inappropriate prescribing directly resulting from unnecessary doses administered. An ability to quantify the proportion of antimicrobial consumption that is unnecessary will allow healthcare organisations and governments to determine targets for a reduction in antimicrobial consumption. This will limit avoidable selection pressure for AMR and can be achieved without compromising patient safety. In addition, whilst PPS can provide a rich data source due to the fact they require a point-in-time assessment, it is not possible to measure the total duration of antibiotic use for patients assessed as part of a PPS. The national audit tool designed for NHS hospitals described in this report aims to support a focussed patient-level assessment of the appropriateness of antibiotic prescribing in secondary care.

It is not currently possible to estimate with any certainty the proportion of antimicrobial prescribing in English hospitals that is inappropriate, as electronic prescribing data directly linking hospital prescriptions to clinical data are still not widely available. A key study aim was to develop and validate a patient-level audit tool that could support estimation of the number of days of antibiotic therapy that auditors considered non-essential and therefore potentially avoidable. The proportion of inappropriate antibiotic prescribing can be represented as non-essential days of therapy (DOTs) expressed as a percentage of total days of therapy. The goal of the audit tool was to collate variables that would support assessment of those aspects of prescribing most relevant for selection pressure and resistance. If successful, this would allow the UK to potentially set goals for reduction of antibiotic consumption in hospitals that are safe and achievable. A secondary aim included creating an audit tool that helped minimise subjectivity in order to standardise the assessment of appropriateness by prompting auditors to consider certain critical information in order to support their decision.

2. Results 2.1. Defining Appropriateness

Defining the gold standard for appropriate antibiotic prescribing is challenging due to the inherently subjective nature of evaluating quality in prescribing. An initial list was compiled to incorporate elements of high-quality prescribing identified by opinion leaders in the field, the UK Government Scientific Advisory Committee on Antimicrobial Prescribing, Resistance and Healthcare Associated Infection (APRHAI) and a workshop including 22 infection experts convened to define appropriate antibiotic prescribing. The collated list (Table 1) sets out many elements of antibiotic prescribing that may be considered in an assessment of appropriateness.

It could reasonably be argued that each of these elements is relevant to efficacy, safety, and improving patient outcomes, and critical to the assessment of appropriateness of prescribing. However, the APRHAI committee took a view that improvement in performance against certain elements would be less likely to impact upon the overall consumption of antibiotics and, consequently, would be less likely to impact upon antibiotic resistance. Three aspects were therefore given ultimate priority as most relevant to identifying avoidable selection of resistance within the UK hospital setting:

1. Prescribing an antibiotic for a patient in the absence of (documented) evidence of bacterial infection.

2. Prescribing a critical broad-spectrum antibiotic to patients in the absence of a (documented) rationale.

3. Continuing an antibiotic prescription beyond the course length recommended in local or national guidelines, in the absence of a (documented) rationale.

2.2. Initial Draft of Audit Tool

A consensus emerged from the one-day workshop that existing antibiotic prescribing guidelines for acute hospitals often do not suggest clinical thresholds for initiating or stopping antibiotics. This means there is a lack of consistency between hospitals in assessing whether antibiotics are indicated and for how long. There was also general agreement that antibiotic consumption data alone were not sufficient to determine the proportion of prescribing that is inappropriate. This was further complicated by the failure of hospital-level antibiotic consumption data to account for patient acuity and hospital case mix. The workshop group arrived at the conclusion that an individual patient audit by an infection specialist was required to establish with greater certainty the current proportion of prescribing that is inappropriate. This process was anticipated to be less vulnerable to case mix or speciality bias. It was agreed that a patient-level audit tool would be sent for expert elicitation via a Delphi process.

2.3. Round 1 Delphi

The response rate to the initial survey was 100% (19/19), taking on average 32 minutes to complete. Of the first-round responders, 42% (8/19) were specialist antimicrobial pharmacists, 21% (4/19) were infectious disease doctors or medical microbiologists, 16% (3/19) were general medical doctors, and the remaining 21% (4/19) were antimicrobial stewardship nurses. The majority (14/19) of participants in round one had over five years’ experience and practice in antimicrobial stewardship and infection. When asked what proportion of inpatients prescribed antibiotics should be audited to provide reasonable confidence of the overall assessment of the appropriateness of antibiotic prescribing in secondary care, 79% (15/19) expressed the view that either 5–10% or 10–25% of inpatients on antibiotics would be an adequate sample size to represent the entire hospital patient population. The panel members were also asked at what frequency would it be necessary and reasonable to repeat an audit assessing appropriate antibiotic prescribing; 89% (17/19) indicated annually and 11% (2/19) stated every three years would be sufficient. Twenty-three of the 25 variables were scored as relevant or highly relevant for assessing appropriateness for one or more of the main decision points (Supplementary Materials, Table S1) and were therefore accepted following Round 1. One variable, “documentation of pre-72-hour review”, was rejected by the expert panel in Round 1 as not relevant to the assessment of appropriateness at any of the main decision time points. A neutral consensus was reached by the expert panel for the variable “presenting complaint”; this was subsequently rescored in the second round. The first variable was renamed to “review of antibiotic prescription within 72 h” to remove the requirement for formal documentation of the review and presented for scoring in the second round. The project team was reluctant to remove this variable due to the potential value it brings in predicting unnecessarily prolonged treatment courses.

2.4. Round 2 Delphi

Out of the nineteen respondents (from Round 1) who were invited to participate in Round 2 the response rate was 68% (13/19). The same distribution of healthcare professionals participated in Round 2 with 69% (9/13) having over 5 years’ experience and practice in antimicrobial stewardship and infection. The neutral variable “presenting complaint” was rescored, with 85% (11/13) of respondents indicating this was considered relevant to assessing appropriate prescribing. A consensus was reached when the expert panel members were asked to score the renamed variable “review of antibiotic prescription within 72 h” with this being accepted by the panel (Figure 1).

The majority (9/13) of respondents reported that there were local guidance or supportive tools available to help clinicians stop antibiotic therapy safely. Where not available, three of the four respondents agreed that such guidance would be helpful.

2.5. Feasibility

Eight of the 19 original panel members agreed that the audit tool was fit for purpose, 32% (6/19) expressed a neutral view, and 5 panel members disagreed (5/19) including one panel member stating they were unable to assess this question. The participants were asked if the time taken to complete the audit tool is a worthwhile investment of NHS resources for the benefit of patient safety and public health, with 43% (8/19) agreeing that it is, 21% (4/19) having a neutral view, 26% (5/19) disagreeing, and 10% (2/19) being unable to assess this.

3. Discussion

An audit tool consisting of 25 variables to support assessment of the appropriateness of antibiotic prescribing in secondary care has been developed. The prototype audit tool was reviewed by 19 multi-disciplinary infection specialist health professionals through a two stage modified Delphi process and all 25 variables were accepted by specialists as relevant or very relevant to the aim of the audit following rewording of just one variable.

The response rate of participants within the Delphi process was seen to reduce between rounds one and two. A systematic review of 31 studies indicated that the median response rate of a Delphi process does reduce by 2% from the first to the last round of Delphi processes [30], however even with reminder e-mails our response reduced by 32% from round one to two, so this could be interpreted as introducing potential bias. The Delphi process occurred during winter months which can operationally be a busy time for NHS hospitals. This could have been a contributing factor in the reduced response rate seen over the two rounds.

The inclusion of acuity scores within the audit tool was to help support the auditor in assessing appropriateness of antibiotics by highlighting the severity of the acute health status which could be as a result of a likely infection. The supportive use is important as a low acuity score does not rule out infection, but merely indicates no severe or deteriorating condition or a lesser probability of organ failure. Evidence from a multi-centre study of adult emergency departments in the UK reported a positive predictive value for National Early Warning Score (NEWS) ≥3 of 20% for mortality or ICU admission [31]. It was therefore considered reasonable to assess what proportion of patients treated with antibiotics either have NEWS ≥3 or localised evidence of infection at an anatomical site. An abbreviated version of the Sequential Organ Failure Assessment (SOFA) score, quick SOFA (qSOFA), was assessed in almost 75,000 adult patients hospitalised with suspected infection [32]. The predictive validity of the qSOFA score for in-hospital mortality was found to be statistically greater than the SOFA or systemic inflammatory response syndrome (SIRS) scores, supporting its use as a prompt to consider possible sepsis [32]. Applying qSOFA score ≥2 during emergency department or ward stay as a prediction tool for in-hospital all-cause mortality had a negative predictive value of 97% in adult patients with clinical suspicion of infection in a recent multi-centre study including 27 French emergency departments [33]. A qSOFA score at a threshold of 2 or more may be considered by some clinicians to be a reasonable tool for differentiating patients in whom immediate broad-spectrum antibiotics are justified, given the associated in-hospital mortality of 24% [33].

During the second round of the Delphi process the panel did not accept that formal documentation of a pre-72-hour review was relevant. The panel did however accept that whether a pre-72-hour review took place was relevant, irrespective of documentation (e.g., prescription changed but no corresponding entry in the medical notes) for the purposes of assessing appropriateness of antibiotic prescribing. In the absence of a pre-72-hour review, it is likely that patients for whom diagnostic tests support an alternative diagnosis to infection are likely to continue antibiotics longer than necessary. This variable is also important due to the potential value it brings to predicting unnecessarily prolonged treatment courses, and it is part of ongoing quality improvement work within English hospitals, so the indicator was reworded for Round 2. Secondary healthcare quality improvement work within England has focussed on reducing total antibiotic consumption as well as obtaining evidence of antibiotic review within 72 h of commencing an antibiotic [34].

The addition of separate questions to explore individual views on inappropriate prescribing within secondary care was used as it was not possible to estimate a sample size, because the proportion of non-essential DOTs was unknown. Delphi participants were asked what proportion of patients on antibiotics required auditing to give them assurance on appropriateness of antibiotic prescribing for their hospital.

The expert panel members were invited to estimate the feasibility and likely value of the audit tool; this saw 26% of the expert panel disagreeing with statements that the audit tool was fit for purpose or a worthwhile investment of NHS resources. This finding was surprising given the high degree of consensus over the relevance of individual variables for supporting the assessment of appropriateness and was not explored in more detail. It may reflect a perception of futility that a simple audit tool could not sufficiently capture the complexity of an episode of infection to enable a reproducible assessment of appropriateness. A pilot study of the audit tool to test the reliability of the instrument is clearly a prerequisite before more widespread adoption.

3.1. Study Strengths and Limitations

Strengths of this project include the development of an audit tool through a consensus of a multi-disciplinary team of experienced infection specialists working in hospital clinical practice. The authors believe the developed audit tool is the first of its kind internationally that can specifically quantify the scale of inappropriate prescribing directly resulting from unnecessary doses being administered. The audit tool was designed to minimise the burden of data collection by limiting the items of data collected to only those considered relevant or highly relevant to the specific aim of evaluation of appropriateness of prescribing at three critical prescribing decision time points. The audit tool also narrows the focus of the audit to evaluate only those elements of prescribing associated with unnecessary antibiotic doses and therefore likely to be most relevant to the selection of resistant microorganisms: whether an antibiotic is indicated at the start of treatment, whether treatment beyond 72 h is justified, and whether treatment beyond a standard course length is appropriate.

The potential to collect more detailed clinical information to explore the subjective decision that the auditors would take was advantageous. Standardisation of the assessment process may be aided by the inclusion of objective variables to capture evidence of infection or sepsis, along with markers of severity including qSOFA score, NEWS, C-reactive protein (CRP), and white blood cell count.

The potential limitations of the audit tool include the time needed to collect data and risk of subjectivity in assessment which are inherent limitations of this approach, however a balance is struck between the value of detailed patient-level data to robustly answer the important question of appropriateness set against the use of a standardised data collection tool to minimise variability in assessment. There was also limited opportunity for experts to consider variables not included in the tool, such as medical imaging and urine dipstick tests. Presenting an increased number of potentially relevant variables such as antibiotic dose and route of administration may have provided greater opportunity to discriminate between the perceived value of different variables for estimating non-essential doses. Whilst a relatively small sample of experts participated in the Delphi process, an important limitation, it is worth noting that 22 experts participated in the workshop to define appropriate antibiotic prescribing and develop the initial audit tool cascaded for the Delphi consensus. Subsequent piloting of the audit tool will need to include ongoing evaluation of the strengths and weakness of the tool and inform iterative improvements. The minor changes to the audit tool as a result of this Delphi process provide reassurance that extensive revision of the tool was not required, however a higher number of experts participating would have provided greater assurance of a reliable rating for each of the variables assessed.

Since admission and discharge dates were not included in the tool it was not possible to determine at which point during their admission a patient receives antibiotics and therefore whether a patient is receiving treatment for a healthcare-associated infection or community-acquired infection unless this is stated explicitly by the auditor. This was identified by the project team as a priority to capture in future versions of the audit tool.

The use of a web-based Delphi process limited opportunity for qualitative feedback from a face-to-face discussion and may have contributed to the lower response rate for Round 2. The selection of expert panel members was via national infection networks and only one healthcare professional also participated in the previous workshop that convened to define appropriate prescribing. To help reduce potential bias, all participants were blinded to who participated in the expert panel with individual correspondence being sent.

The use of the developed audit tool is for active treatment and does not assess long-term and peri-operative surgical prophylaxis antibiotics. An adaptation could be made to the audit tool in order to assess these groups of patients. Finally, the developed audit tool was prioritised for antibacterial agents only but could also be adapted for antifungal and antiviral agents.

3.2. Future Work

Piloting the audit tool within NHS hospitals to assess usability and operability is the next step. This would also provide additional insight and development of the tool and mitigate against the limitations of the small consensus panel. The resource burden of data collection and whether there is variation between hospitals that use electronic systems or paper-based processes needs to be quantified. Although it is anticipated that the use of electronic systems should be faster than paper-based prescribing, ease of access to information for each variable may vary. Piloting the audit tool within 5 to 10 hospitals could provide a preliminary assessment of non-essential days of antibiotic therapy prescribed to inform the sample size calculation for a definitive study. Recruiting a mixture of hospital types (teaching, district and specialist) will be important to assess the practicality of the developed audit tool. The balance between the amount of information being collected with the time taken to collect the data in a busy operational hospital will be crucial.

4. Materials and Methods 4.1. Part 1: Development of the Audit Tool

Evidence synthesis of published literature and international guidance contributed to the development of the initial prototype audit tool to try and quantify and identify inappropriate antimicrobial prescribing. The initial prototype audit tool then underwent preliminary feasibility testing at one acute teaching hospital in Southampton, England. This was conducted by a specialist pharmacist supported by a medical microbiologist, with a focus on evaluating the operability of the audit tool. After a subgroup of the UK government APRHAI scientific advisory committee drafted standards to define appropriate prescribing in secondary care a finalised prototype audit tool was created [4]. The subgroup discussed aspects of auditing including frequency, intensity, resourcing, and mechanism of data collection including feedback and reporting processes.

To elicit further expert opinion, a one-day workshop was held on 6 March 2017 with infection experts including secondary care clinicians (10/22), pharmacists (9/22), a nurse, an academic, and a medical director to explore views on the appropriateness of prescribing within hospitals. The project team including two pharmacists of the antimicrobial resistance programme at Public Health England and three national antimicrobial pharmacists met to discuss the workshop findings and updated the audit tool accordingly. Elements of the audit tool that were intended to support the assessment of appropriateness of antimicrobial prescribing were termed “variables” for the purposes of the validation process. The selection of variables to include in the audit tool was based on discussions within the antibiotic prescribing quality subgroup of APRHAI, iterative e-mail feedback from co-authors, and prototype audit tool testing in Southampton.

To support the auditor in their assessment of the severity of infection and deterioration as a result of a likely infection the National Early Warning Score (NEWS) and quick sequential organ failure assessment (qSOFA) score were incorporated into the audit tool [32,35]. A threshold NEWS value of 3 or above has been proposed as a suitable trigger to systematically screen for sepsis or septic shock in patients with suspected infection in a United States Emergency Department (ED) population with a reported negative predictive value of 99.5 (95% CI 97.8–99.9) [36].

4.2. Part 2: Validation of the Audit Tool

Between October and December 2017, a RAND-modified Delphi process was conducted to validate each individual variable within the audit tool. A multi-disciplinary expert panel of infection and public health specialists was recruited via e-mail through national infection and microbiology networks and asked to assess the relevance of each variable against a five-point Likert Scale [37]. The relevance was assessed in relation to three main decision time points during a course of antibiotic therapy:

1. Initiation (was the antibiotic indicated and necessary at the start date?);

2. Early post prescription review (was the antibiotic continued after infection was ruled out?);

3. End of therapy (was the antibiotic continued beyond the standard duration?).

Non-essential days of antibiotic therapy were included with potential reasons as follows: (1) antibiotic not indicated/unnecessary at start date; (2) unexplained continuation of antibiotic after infection ruled out; and (3) unexplained continuation of antibiotic beyond standard duration. The standard duration was defined by individual hospitals as identified in local antibiotic guidelines. A separate questionnaire was sent to the Delphi participants to explore individual views on inappropriate prescribing within secondary care.

The expert panel participated in two rounds. Invitation via e-mail was sent to 19 participants before the first round. The online survey SelectSurvey.net was used in each Delphi round with participant reminder e-mails sent periodically. The results were discussed within the project team after the first round and the aggregated scores were shared with the participants prior to the second round. Each of the 25 variables was assessed for its relevance against the three main decision time points. Qualitative feedback was reviewed by one project team member and common themes reported for discussion with the full project team. The threshold for accepting a variable was met if the median score was “relevant” or “very relevant” for assessing appropriateness at any one of the three decision time points. A variable was rejected if the median score for all three of the critical decision points was “irrelevant” or “very irrelevant” with a neutral score being rescored in the second round to provide a definitive response. The rewording of questions between each round occurred by the project team and was presented to the participants from Round 1 for Round 2. After completing both rounds of the Delphi process the accepted variables were incorporated into a finalised audit tool (Supplementary Materials, Figure S1).

4.3. Statistics

Data were summarised using non-parametric descriptive statistics which are considered suitable to measure consensus and stability in Delphi studies [38] and due to the number of Delphi participants. The median and percentage “relevant” and “highly relevant” responses for each of the variables in relation to each decision time point was calculated using univariate analysis.

5. Conclusions

In summary, the RAND-modified Delphi process has provided face validity of this audit tool and selection of critical variables that can be used to support the assessment of inappropriate antibiotic prescribing. The feedback from expert panel members helped shape development of the audit tool and provides an important endorsement from practising healthcare professionals within and outside the NHS. This audit methodology will help hospitals identify the location, extent, and nature of inappropriate antibiotic prescribing and provide evidence to drive local antimicrobial stewardship and focus targeted training to specific areas of improvement.

Supplementary Materials

The following are available online at https://www.mdpi.com/2079-6382/8/2/49/s1. Figure S1: Assessing appropriateness of antibiotic therapy–final national audit tool for pilot. Table S1: Round 1 Delphi results.

Author Contributions

Conceptualization, D.A.-O., G.H., K.S.H.; data curation, G.H.; formal analysis, G.H.; methodology, G.H., K.S.H., and D.A.-O.; project administration, G.H.; supervision, D.A.-O; validation, K.S.H., E.C., and D.A.-O.; writing—original draft, G.H.; writing—review & editing, G.H., K.S.H., E.C., P.H., S.H., and D.A.-O.

Funding

This research received no external funding and was conducted by Public Health England

Acknowledgments

The authors would like to thank all the expert panel members who participated in the Delphi process who contributed to the development of the audit tool: Janet Appleton, Druin Burch, Kay Cawthron, Abid Hussain, Conor Jamieson, David Jenkins, Alasdair MacGowen, Sanjay Patel, Neil Powell, Jacqueline Sneddon, Adel Sheikh, Paul Wade, Laura Whitney, Antony Zorzi, James Hatcher, Enrique Castro-Sanchez, Emily Bennett, Cairine Gormley and Tania Misra.

Conflicts of Interest

The authors declare no conflict of interest.

References

1. WHO Antimicrobial Resistance. Available online: http://www.who.int/mediacentre/factsheets/fs194/en/ (accessed on 17 January 2019).

2. Review on Antimicrobial Resistance. Available online: https://amr-review.org/background.html (accessed on 17 January 2019).

3. English Surveillance Programme for Antimicrobial Utilisation and Resistance (ESPAUR) Report 2018. Available online: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/ attachment_data/file/759975/ESPAUR_2018_report.pdf (accessed on 22 March 2019).

4. English Surveillance Programme for Antimicrobial Utilisation and Resistance (ESPAUR) Report 2017. Available online: https://assets.publishing.service.gov.uk/government/uploads/system/uploads/ attachment_data/file/656611/ESPAUR_report_2017.pdf (accessed on 18 January 2019).

5. Point Prevalence Survey of Healthcare-Associated Infections and Antimicrobial Use in European Acute Care Hospitals Surveillance Report 2011–2012. Available online: https://ecdc.europa.eu/sites/portal/files/media/en/publications/Publications/healthcare-associated-infections-antimicrobial-use-PPS.pdf (accessed on 18 January 2019).

6. Kieran, J.A.; O’Doherty, R.G.; Hudson, B. ESAC point prevalence methodology to assess antimicrobial consumption and quality of prescribing in an Australian setting. Med. J. Aust. 2011, 194, 103–104.

7. James, R.; Upjohn, L.; Cotta, M.; Luu, S.; Marshall, C.; Buising, K.; Thursky, K. Measuring antimicrobial prescribing quality in Australian hospitals: Development and evaluation of a national antimicrobial prescribing survey tool. J. Antimicrob. Chemother. 2015, 70, 1912–1918.

8. Spivak, E.S.; Cosgrove, S.E.; Srinivasan, A. Measuring appropriate antimicrobial use: Attempts at opening the black box. Clin. Infect. Dis. 2016, 63, 1639–1644.

9. Lopez-Lozano, J.M.; Monnet, D.L.; Yague, A.; Burgos, A.; Gonzalo, N.; Campillos, P.; Saez, M. Modelling and forecasting antimicrobial resistance and its dynamic relationship to antimicrobial use: A time series analysis. Int..J Antimicrob. Agents 2000, 14, 21–31.

10. Tacconelli, E.; Cataldo, M.A.; De Pascale, G.; Manno, D.; Spanu, T.; Cambieri, A.; Antonelli, M.; Sanguinetti, M.; Fadda, G.; Cauda, R. Prediction models to identify hospitalized patients at risk of being colonized or infected with multidrug-resistant Acinetobacter baumannii calcoaceticus complex. J. Antimicrob. Chemother. 2008, 62, 1130–1137. [Green Version]

11. Tacconelli, E.; De Angellis, G.; Cataldo, M.A.; Mantengoli, E.; Spanu, T.; Pan, A.; Corti, G.; Radice, A.; Stolzuoli, L.; Antinori, S.; et al. Antibiotic usage and risk of colonization and infection with antibiotic-resistant bacteria: A hospital population-based study. Antimicrob. Agents Chemother. 2009, 53, 4264–4269.

12. Braykov, N.P.; Morgan, D.J.; Schweizer, M.L.; Usian, D.Z.; Kelesidis, T.; Weisenberg, S.A.; Johannsson, B.; Young, H.; Cantey, J.; Scrinivason, A.; et al. Assessment of empirical antibiotic therapy optimisation in six hospitals: An observational cohort study. Lancet Infect. Dis. 2014, 14, 1220–1227.

13. De Sousa, A.G.; Fernandes Junior, C.J.; Santos, G.P.D.; Laselva, C.R.; Polessi, J.; Lisboa, L.F.; Akamine, N.; Silva, E. The impact of each action in the Surviving Sepsis Campaign measures on hospital mortality of patients with severe sepsis/septic shock. Einstein 2008, 6, 323–327.

14. Paul, M.; Shani, V.; Muchtar, E.; Kariv, G.; Robenshtok, E.; Leibovici, L. Systematic review and meta-analysis of the efficacy of appropriate empiric antibiotic therapy for sepsis. Antimicrob. Agents Chemother. 2010, 54, 4851–4863.

15. Harris, A.D.; Perencevich, E.; Roghmann, M.C.; Morris, G.; Kaye, K.S.; Johnson, J.A. Risk factors for piperacillin-tazobactam-resistant Pseudomonas aeruginosa among hospitalized patients. Antimicrob. Agents Chemother. 2002, 46, 854–858.

16. Lai, C.C.; Wang, C.Y.; Chu, C.C.; Tan, C.K.; Lu, C.L.; Lee, Y.C.; Huang, Y.T.; Lee, P.I.; Hsueh, P.R. Correlation between antibiotic consumption and resistance of Gram-negative bacteria causing healthcare-associated infections at a university hospital in Taiwan from 2000 to 2009. J. Antimicrob. Chemother. 2011, 66, 1374–1382. [Green Version]

17. Pakyz, A.L.; Oinonen, M.; Polk, R.E. Relationship of carbapenem restriction in 22 university teaching hospitals to carbapenem use and carbapenem-resistant Pseudomonas aeruginosa. Antimicrob. Agents Chemother. 2009, 53, 1983–1986.

18. Opatowski, L.; Mandel, J.; Varon, E.; Boelle, P.Y.; Temime, L.; Guillemot, D. Antibiotic dose impact on resistance selection in the community: A mathematical model of beta-lactams and Streptococcus pneumoniae dynamics. Antimicrob. Agents Chemother. 2010, 54, 2330–2337.

19. Guillemot, D.; Carbon, C.; Balkau, B.; Geslin, P.; Lecoeur, H.; Vauzelle-Keroedan, F.; Bouvenot, G.; Eschwege, E. Low dosage and long treatment duration of beta-lactam: Risk factors for carriage of penicillin-resistant Streptococcus pneumoniae. JAMA 1998, 279, 365–370.

20. Handel, A.; Margolis, E.; Levin, B.R. Exploring the role of the immune response in preventing antibiotic resistance. J. Theor. Biol. 2009, 256, 655–662.

21. Martinez, M.N.; Papich, M.G.; Drusano, G.L. Dosing regimen matters: The importance of early intervention and rapid attainment of the pharmacokinetic/pharmacodynamic target. Antimicrob. Agents Chemother. 2012, 56, 2795–2805.

22. Olofsson, S.K.; Cars, O. Optimizing drug exposure to minimize selection of antibiotic resistance. Clin. Infect. Dis. 2007, 45 (Suppl. 2), S129–S136.

23. Schrag, S.J.; Pena, C.; Fernandez, J.; Sanchez, J.; Gomez, V.; Perez, E.; Feris, J.M.; Besser, R.E. Effect of short-course, high-dose amoxicillin therapy on resistant pneumococcal carriage: A randomized trial. JAMA 2001, 286, 49–56.

24. Tam, V.H.; Louie, A.; Deziel, M.R.; Liu, W.; Drusano, G.L. The relationship between quinolone exposures and resistance amplification is characterized by an inverted U: A new paradigm for optimizing pharmacodynamics to counterselect resistance. Antimicrob. Agents Chemother. 2007, 51, 744–747.

25. Singh, N.; Rogers, P.; Atwood, C.W.; Wagener, M.M.; Yu, V.L. Short-course empiric antibiotic therapy for patients with pulmonary infiltrates in the intensive care unit. A proposed solution for indiscriminate antibiotic prescription. Am. J. Respir. Crit. Care Med. 2000, 162, 505–511.

26. Chastre, J.; Wolff, M.; Fagon, J.Y.; Chevret, S.; Thomas, F.; Wermert, D.; Clementi, E.; Gonzalez, J.; Jusserand, D.; Asfar, P.; et al. PneumA trial group. comparison of 8 vs 15 days of antibiotic therapy for ventilator-associated pneumonia in adults: A randomized trial. JAMA 2003, 290, 2588–2598.

27. Palacios-Baena, Z.R.; Gutierrez-Gutierrez, B.; De Cueto, M.; Viale, P.; Venditti, M.; Hernandez-Torres, A.; Oliver, A.; Martinez-Martinez, L.; Calbo, E.; Pintado, V.; et al. REIPI/ESGBIS/INCREMENT Group. Development and validation of the INCREMENT-ESBL predictive score for mortality in patients with bloodstream infections due to extended-spectrum-beta-lactamase-producing Enterobacteriaceae. J. Antimicrob. Chemother. 2017, 72, 906–913.

28. Schuts, E.C.; Hulscher, M.E.; Mouton, J.W.; Verduin, C.M.; Stuart, J.W.T.C.; Overdiek, H.W.P.M.; van der Linden, P.D.; Natsch, S.; Hertogh, C.M.P.M.; Wolfs, T.F.W.; et al. Current evidence on hospital antimicrobial stewardship objectives: A systematic review and meta-analysis. Lancet Infect. Dis. 2016, 16, 847–856.

29. Marra, A.R.; de Almeida, S.M.; Correa, L.; Silva, M., Jr.; Martino, M.D.; Silva, C.V.; Cal, R.G.; Edmond, M.B.; dos Santos, O.F. The effect of limiting antimicrobial therapy duration on antimicrobial resistance in the critical care setting. Am. J. Infect. Control. 2009, 37, 204–209.

30. Boulkedid, R.; Abdoul, H.; Loustau, M.; Sibony, O.; Alberti, C. Using and reporting the delphi method for selecting healthcare quality indicators: A systematic review. PLoS ONE 2011, 6, e20476.

31. Corfield, A.R.; Lees, F.; Zealley, I.; Houston, G.; Dickie, S.; Ward, K.; McGuffie, C. Utility of a single early warning score in patients with sepsis in the emergency department. Emerg. Med. J. 2014, 31, 482–487.

32. Seymour, C.W.; Liu, V.X.; Iwashyna, T.J.; Iwashyna, T.; Brunkhorst, F.M.; Rea, T.D.; Scherag, A.; Rubenfeld, G.; Kahn, J.M.; Shankar-Hari, M.; et al. Assessment of clinical criteria for sepsis: For the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA 2016, 315, 762–774.

33. Freund, Y.; Lemachatti, N.; Krastinova, E.; Van Laer, M.; Claessens, Y.-E.; Avonso, A.; Occelli, C.; Feral-Pierssens, A.-L.; Truchot, J.; Ortega, M.; et al. Prognostic accuracy of sepsis-3 criteria for in-hospital mortality among patients with suspected infection presenting to the emergency department. JAMA 2017, 317, 301–308.

34. NHS England National CQUIN Templates 2016/17. Available online: https://www.england.nhs.uk/nhs-standard-contract/cquin/cquin-16-17/ (accessed on 18 January 2019).

35. National Early Warning Score (NEWS): Standardising the Assessment of Acute Illness Severity in the NHS. Report of a Working Party. Available online: https://www.rcplondon.ac.uk/projects/outputs/national-early-warning-score-news (accessed on 18 January 2019).

36. Keep, J.W.; Messmer, A.S.; Sladden, R.; Burrell, N.; Pinate, R.; Tunnicliff, M.; Glucksman, E. National early warning score at Emergency Department triage may allow earlier identification of patients with severe sepsis and septic shock: A retrospective observational study. Emerg. Med. J. 2016, 33, 37–41.

37. Vagias, W.M. Likert-Type Scale Response Anchors; Clemson International Institute for Tourism & Research Development, Department of Parks, Recreation and Tourism Management, Clemson University: Clemson, SC, USA, 2006.

38. Holey, E.A.; Feeley, J.L.; Dixon, J.; Whittaker, V.J. An exploration of the use of simple statistics to measure consensus and stability in Delphi studies. BMC Med. Res. Methodol. 2007, 7, 52.