Essential notion of the health economic evaluation: Definition

Abstract

Economic evaluation (EE) is a dynamically advancing knowledge area of health economics. It has been conceived to provide evidence for allocating scarce resources to gain the best value for money. The problem of efficiency of investments becomes even more crucial with advances in modern medicine and public health which bring about both improved patient outcomes and higher costs. Despite the abundance of literature on the EE concepts, some key notions including the definition of the economic evaluation remain open for discussion. Academic literature offers a large number and growing variety of EE definitions. It testifies to the fact that existing definitions do not meet economists' requirements and they choose to create new definitions. The aim of this study is to explore existing definitions, identify knowledge gaps and formulate a comprehensive EE definition. Several methodologies were used to achieve the research objective: identification of related peer-reviewed papers using Google Scholar and EBSCO databases (e.g. MEDLINE Complete, CINAHL Complete, Health Business Elite, Health Policy Reference Center, Business Source Complete), critical literature review, critical thinking and inductive reasoning. The study identified and examined 60 existing non-identical EE definitions, revealed their common focus points and grouped them into 8 clusters. Revealed common focus points were compared with the widely-accepted EE definition by Professor Michael Drummond and integrated in a comprehensive EE definition. A new, comprehensive definition of EE has been proposed.

Headnote

Abstract

Background

Economic evaluation (EE) is a dynamically advancing knowledge area of health economics. It has been conceived to provide evidence for allocating scarce resources to gain the best value for money. The problem of efficiency of investments becomes even more crucial with advances in modern medicine and public health which bring about both improved patient outcomes and higher costs. Despite the abundance of literature on the EE concepts, some key notions including the definition of the economic evaluation remain open for discussion. Academic literature offers a large number and growing variety of EE definitions. It testifies to the fact that existing definitions do not meet economists' requirements and they choose to create new definitions.

Methods

The aim of this study is to explore existing definitions, identify knowledge gaps and formulate a comprehensive EE definition. Several methodologies were used to achieve the research objective: identification of related peer-reviewed papers using Google Scholar and EBSCO databases (e.g. MEDLINE Complete, CINAHL Complete, Health Business Elite, Health Policy Reference Center, Business Source Complete), critical literature review, critical thinking and inductive reasoning.

Results

The study identified and examined 60 existing non-identical EE definitions, revealed their common focus points and grouped them into 8 clusters.

Conclusions

Revealed common focus points were compared with the widely-accepted EE definition by Professor Michael Drummond and integrated in a comprehensive EE definition. A new, comprehensive definition of EE has been proposed.

Resumen

Fondo

La evaluación económica (EE) es un área de conocimiento que avanza dinámicamente en la economía de la salud. Se ha concebido para proporcionar pruebas de la asignación de recursos escasos para obtener la mejor relación calidad-precio. El problema de la eficiencia de las inversiones se vuelve aún más crucial con los avances en la medicina moderna y la salud pública que traen tanto mejores resultados de los pacientes y mayores costos. A pesar de la abundancia de literatura sobre los conceptos de EE, algunas nociones claves incluyendo la definición de la evaluación económica permanecen abiertas para discusión. La literatura académica ofrece un gran número y variedad creciente de definiciones de EE. Prueba del hecho de que las definiciones existentes no cumplen los requisitos de los economistas y optan por crear nuevas definiciones.

Métodos

El objetivo de este estudio es explorar las definiciones existentes, identificar brechas de conocimiento y formular una definición global de EE. Se utilizaron varias metodologías para lograr el objetivo de la investigación: identificación de artículos relacionados revisados por pares utilizando las bases de datos de Google Scholar y EBSCO (por ejemplo, MEDLINE Complete, CINAHL Complete, Health Business Elite, Health Policy Reference Center, Business Source Complete) revisión crítica de la literatura, pensamiento y razonamiento inductivo.

Resultados

El estudio identificó y examinó 60 definiciones no idénticas de EE, reveló sus puntos comunes de enfoque y los agrupó en 8 grupos.

Conclusiones

Los puntos de enfoque común revelados se compararon con la definición de EE ampliamente aceptada por el profesor Michael Drummond y se integraron en una definición comprensiva de EE. Se ha propuesto una nueva y amplia definición de EE.

JEL Classification: D61, I18, I100, I110, O3

Keywords: Economic evaluation, health care, definition, evidence-informed policy, health technology assessment, health economics

Abbreviations

EE: economic evaluation NICE: National Institute for Health and Clinical Excellence

Declarations

Acknowledgment

The views, opinions and conclusions expressed in this document are those of the author alone and do not necessarily represent the views of his current or former employer(s) or organizations he is affiliated with.

Funding

The author received no funding.

Conflict of Interest

There is no conflict of interest.

Ethics approval and consent to participate

This research, based entirely on available public data, did not require approval of an ethics committee.

Consent for publication

Not applicable.

Availability of data and material

Data sharing not applicable to this article as no datasets were generated or analysed during the current study

Competing interests

The author declares that there are no competing interests.

1.- Background

"The beginning of wisdom is the definition of terms.

Attributed to Socrates (470 - 339 B.C.)

Advances in modern medicine and public health bring about both improved patient outcomes and higher costs. Health care spending is growing in many countries. Projected US health annual spending growth for 2014-24 is to average 5.8 percent, raising the health share of US gross domestic product to 19.6 percent in 2024 [1].

Economic evaluations (EE) have been conceived to provide evidence for allocating scarce resources to gain the best value for money. Through its several decades of development, EE turned into an established multidisciplinary area of knowledge which lies on the intersection of economics, medicine, health care, evaluation research, outcomes research, comparative effectiveness research, health technology assessment, etc.

EE is a dynamically advancing area of health economics (e.g. [2]). Increasingly large number of EEs is being conducted and reported. Recent bibliographic analysis of the EE publications confirms global interest to and large volume of publications on the subject [3]. Pitt and colleagues identified 2,844 unique, complete EEs published during a period from 1 January 2012 to 3 May 2014 (28 months). These EEs were published in a total of 967 different journals (although 75% of these journals published no more than 1-2 articles). Most intensive research is focused on five areas: cancer and other neoplasms, cardiovascular diseases, mental health, cognition, and developmental and behavioural disorders (including self-harm and substance disorders), musculoskeletal diseases (including back pain) and respiratory diseases - comprising over 60% of total EEs. In each of these health areas one to two hundred EEs published annually. EEs were also conducted in another 20 health areas. Average annual number of complete EEs was observed to be 1,219 (in 2012-14).

Even though the numbers of the EEs conducted around the world are impressive, most likely they are not all-inclusive. One of the reasons is a variability of the terminology used. In some cases, EEs are referred to as economic appraisals (e.g. [4]), technology appraisals (e.g. [5]) or economic assessments (e.g. [6]).

Despite the abundance of literature on the EE concepts, one of the key notions - EE definition - still captures interest of researchers. This is evidenced by multiple peer-reviewed papers offering a variety of EE definitions.

2.- Methods

The purpose of this study is to explore existing definitions, identify knowledge gaps and formulate a comprehensive EE definition.

Several methodologies were used to achieve the research objective: identification of related peerreviewed papers, critical literature review, critical thinking and inductive reasoning. The search was conducted in Google Scholar and several databases through the EBSCO integrated search including MEDLINE Complete, CINAHL Complete, Health Business Elite, Health Policy Reference Center, BioMed Central, Business Source Complete, PubMed, The Cochrane Library, etc.

The paper is structured as follows. Next section provides a literature review of multiple EE definitions. It follows by a section that is focused on the analysis of existing definitions, discuses observations and formulates a proposed definition. Concluding remarks finalize the paper and set directions for future research.

3.- Results

A widely accepted definition of economic evaluation is given by Drummond M.F. and his coauthors:

"[Economic evaluation -] the comparative analysis of alternative courses of action in terms of both their costs and consequences." [7].

This definition (referred to as Drummond's definition below) can be found in all editions (first - 1987 through fourth - 2015) of a very popular book authored by Drummond, et al. Methods for the economic evaluation of health care programmes. Also, it can be traced back even earlier to a work by McMaster University professor Stoddart [8] (one of the co-authors of the book), (as quoted in [9]). Drummond's definition has been cited in numerous papers (e.g. [10, 11, 12]). Many authors adapted Drummond's definition with minimal changes (e.g. [4, 13, 14, 15, 16, 17, 18]). A number of other EE definitions listed below clearly have been influenced by the Drummond's wording.

A plethora of healthcare economic evaluation definitions may be found in academic literature. Using Google Scholar search engine, we found 60 non-identical definitions of health economic evaluation [19]. Complete wording of the revealed economic evaluation definitions with references is available [19]. The authors are attempting to reveal most important characteristics of the EE and combine them in a compact wording. Some definitions are intended to fully describe the notion, others focus only on certain aspects relevant to the authors' studies (e.g. highlighting EEs purpose, intention, goals or objectives). We analyzed and systematized all definitions based on the common features they demonstrate - termed focus points. A total of 8 groups were identified [19] and are presented below.

First. Several authors emphasize that EE is based on the foundational principles of welfare economics. Specific economic concepts and terms are used in the definitions such as opportunity costs, resource consumption and production, monetary value, level of input relative to the level of output etc.

Second. A clear trend of many definitions is an inclusion of an overarching EE goal - to provide information on which technologies will maximize value for money in healthcare.

Third. In some definitions, similar idea is expressed using a concept of economic efficiency.

Fourth. A number of authors emphasize the intended purpose of the EE - to help policy and decision makers with economic evidence (as one of several types of inputs). This approach can be considered an element of systems thinking. Conducting EE and presenting a report is not an end-point. EE is a component of a larger system and outputs of the EE are intended for further analysis by policy and decision makers. Two important implications follow. First, results of the EE should contain economic evidence which fits (by content and format) the needs of the decision makers to have chances for implementation. Second, results of the EE (however crucial they are) constitute only one of the inputs (types of evidence) when decisions are made. Other inputs include quality, equity, public interest, policy preferences, etc.

Fifth. A number of authors point to an important characteristic of EE - its structured and systematic approach. Official agencies in many countries issued regulatory EE guidelines which determine mandatory requirements to the process of conducting EEs and format of presenting results.

Sixth. It is common for the authors to refer to EE as a framework (container or umbrella) concept encompassing a set of tools, methods or techniques for gathering and processing standardized and quantitative data.

Seventh. Some researchers define EE through detailing the steps of the evaluation process: identification, measurement, valuation and comparison of costs and outcomes.

Eighth. Some definitions introduce EE through a list of methods involved in the framework, e.g. costbenefit analysis, cost-effectiveness analysis, cost-utility analysis, etc.

To conclude the review of the EE definitions, it should be noted that in almost all analyzed groups an attempt was made to add healthcare-specific flavour to definitions. That is achieved by using such terms/concepts as health economic evaluation, interventions, health effects, etc.

4.- Discussion and a new definition proposal

The advantages of the Drummond's definition are in its simplicity and conciseness. At the same time, the definition does not capture many aspects of the EE and has not been changed for over three decades despite the improvements in the health economics body of knowledge and multiple enhancements in the evaluation methods. That may be the reason that many scholars preferred to develop their own definitions presented in the previous section.

Analysis of the Drummond's definition through the lens of the focus points found in the current academic papers on EE raises some questions about its comprehensiveness and shows certain areas of potential improvements.

First, the definition is very generic and is not indicating that it applies to the healthcare evaluations, although evaluations in other sectors (e.g. defence, environment, transportation, education, etc.) have certain differences compared to healthcare. The subject of the definition should be stated - health economic evaluation.

Second, similar to a previous point, the definition uses generic terms "courses of actions" with no relation to the health context. In health care, EE are being applied to a wide range of products and services such as health laws and programs, health policy initiatives and implementation strategies, preventive, diagnostic and treatment interventions, medical devices, drugs, health information systems, etc., e.g. [16, 20, 21,22]. Most frequently used actions should be mentioned in the definition.

Third, the definition is not mentioning the overarching EE goal - optimise allocation of resources to achieve maximum efficiency (best value for money).

Fourth, the definition does not place emphasis on the economic notions such as efficiency, resource inputs, values, etc. It should be noted that, in economics terms, EE examines healthcare goods and services and operates with their inputs (resources) and costs associated with them in relation to outcomes (benefits) and values associated with them.

Fifth, the family (genus) component of the definition refers to EE as an "analysis". This is not precise because EE activities include both analysis and synthesis of evidence. See, for example, chapter 10 of [7]. Also, a variety of relatively new (for economic evaluation) scientific methods are regularly employed in EE, e.g., agent-based modelling [23], decision-analytic modelling [24], qualitative methodologies [25, 26, 27] sophisticated statistical techniques [28, 29, 30 such as Monte Carlo simulation [31], Bayesian methods [32, 33], etc. In our view, EE genus should be characterized as an analytic research with mentioning its data intensive nature.

Sixth, the definition does not highlight that EE comprises a set of methods (not a single method). Actually, EE is a structured and systematic framework of methods (sometimes referred to as a set of tools [34], an umbrella [35] or a container [36] concept.

Seventh, the definition does not clarify the main steps in EE application such as identification, measurement, valuation, comparison, analysis and reporting of costs and outcomes.

Eighth, the definition does not indicate how and by whom the results of the EEs will be used. The output of the EE studies is not a final result - it is intended to provide economic evidence to inform decision- and policy-makers, and EEs' content and presentation should meet the needs of the end-users. This should be stated in the definition.

Finally, it should be noted that EEs can be classified into two categories: full (also called complete or comprehensive) economic evaluations and partial economic evaluations. To be considered a full EE, evaluation must meet strict requirements: both costs and outcomes must be analyzed and, at the same time, two or more alternative interventions must be compared. Cost-effectiveness analysis, cost-utility analysis and cost benefit analysis are the types of full economic evaluation (if applied properly). Only full EEs provide information on economic efficiency. Partial EEs do not meet the above requirements. They either consider costs and/or outcomes of only one intervention (no comparison), or deal with costs only (outcomes are not included). Partial EEs include, for example, cost-of-illness (burden-of-disease), program cost, cost-outcome methods [36, 37, 38, 39]. Partial EEs can be methodologically sound. For example, National Institute for Health and Clinical Excellence (NICE) deals with single technology assessments on a continuous basis and published a separate guide on their appraisal process [40]. Partial EEs provide additional useful economic evidence to decision makers (confirmed by a stream of publications in academic literature).

The Drummond's definition applies only to full EE. It explicitly excludes partial EEs by using terms "both" (meaning costs and outcomes) and "the comparative analysis of alternative" interventions. Theoretically, this approach is understandable because economics, generally, concerns itself with choices. Practically, this requirement will limit (unjustifiably, in our view) the scope of EEs and exclude the whole stream of existing evaluations. That is the reason we are not including mandatory full EE qualifiers in our definition (despite the lack of efficiency perspective).

Comparative analysis of the Drummond's definition and multiple other EE definitions revealed areas of potential improvement. A list below shows what questions a new definition should address (based on the identified focus points) and quick highlights of answers to these questions.

1. What are we concerned with in EE? - value for money

2. What is the subject of the definition? - health economic evaluation

3. What is the family (genus) component of the definition? - analytic research process

4. What are the stages of the process that we define? - identification, measurement...

5. In economic sense, what relationship we examine? - inputs vs outcomes

6. In economic sense, what types of goods are involved? - healthcare goods and services

7. What is the output of the process? - economic evidence

8. Who are the intended users of the output? - decision and policy makers

9. What is the objective we want to contribute to? - improvement of investment efficiency

A proposed wording of the EE definition follows. Special formatting has been applied to facilitate reading and comprehension.

Focusing on the optimal allocation of resources with the best value for money,

Health Economic Evaluation is a data intensive analytic research process,

based on a structured framework of methods for systematic identification, measurement, valuation, comparison, analysis and reporting of

inputs (resources) and costs associated with them in relation to outcomes (benefits) and values associated with them,

regarding healthcare goods and services (ranging across health programs, preventive, diagnostic and treatment interventions, medical devices, drugs),

which is conducted with an objective to provide economic evidence to inform decision and policy makers in order to improve investment efficiency.

This research has the following assumption and limitations.

Assumption. This research is based on an assumption of rational behaviour of economists. It means that economists choose to create a new EE definition, if existing definitions do not meet their requirements and they add or change certain features (focus points) to better describe the notion of EE. Multiple mentioning of the same features by different authors provides evidence of the usefulness of these features for defining the concept.

Limitations. The focus of the paper is on the definition that would apply to the EE as a whole not to specific EE methods (e.g. cost-effectiveness, cost-benefit, etc.). Method-specific definitions were excluded from the analysis. Intervention-specific definitions were excluded from the analysis, e.g. [41].

5.- Conclusion

EE is a dynamically advancing knowledge area and its definition required an update. We analyzed 60 non-identical EE definitions, and grouped them into eight (8) clusters with common focus points. Revealed common focus points were integrated in a comprehensive EE definition. A new, comprehensive definition of EE has been proposed.

References

References

1. Keehan SP, Cuckler GA, Sisko AM, Madison AJ, Smith SD, Stone DA, Poisal JA, Wolfe CJ, Lizonitz JM. National health expenditure projections, 2014-24: spending growth faster than recent trends. Health Affairs, 2015;34(8): 1407-1417. DOI: 10.1377/hlthaff.2015.0600

2. Hutton J. 'Health Economics' and the evolution of economic evaluation of health technologies. Health economics. 2012;21(1): 13-18.

3. Pitt C, Goodman C, Hanson K. Economic evaluation in global perspective: A bibliometric analysis of the recent literature. Health Economics. 2016;25, S1: 9-28. DOI: 10.1002/hec.3305

4. Bowling A. Research methods in health: investigating health and health services. McGraw-Hill Education (UK), 2014.

5. National Institute for Health and Clinical Excellence (NICE). Guide to the methods of technology appraisal 2013.

6. Toumi M, Remuzat C, El Hammi E, Millier A, Aballéa S, Chouaid C, Falissard B. Current process and future path for health economic assessment of pharmaceuticals in France. Journal of Market Access & Health Policy. 2015; 3: 24966. http://dx.doi.org/10.3402/imahp.v3.24966

7. Drummond MF, Sculpher MJ, Claxton K, Stoddart GL, Torrance GW. Methods for the economic evaluation of health care programmes. Oxford university press. 2015.

8. Stoddart GL. On determining the efficiency of health programs. Hamilton: McMaster University, mimeo, November 1980.

9. Kaminsky BA. Resource Allocation in a Regional Structure for the BC Ministry of Health (Doctoral dissertation, University of British Columbia). 1982. https://open.library.ubc.ca/cIRcle/collections/ubctheses/831/items/1.0095267

10. Shiell A, Donaldson C, Mitton C, Currie G. Health economic evaluation. Journal of epidemiology and community health. 2002;56(2): 85-88.

11. Shemilt I, Mugford M, Byford S, Drummond M, Eisenstein E, Knapp M, Mallender J, McDaid D, Vale L, Walker D. Incorporating economics evidence. Cochrane handbook for systematic reviews of interventions. 2008. pp.449-480.

12. Brouwer R, Georiou S. (2012). Economic Evaluation. Chapter 12 in 2012 World Health Organization (WHO). Animal Waste, Water Quality and Human Health. Edited by Al Dufour, Jamie Bartram, Robert Bos and Vic Gannon. 2012. ISBN: 9781780401232. Published by IWA Publishing, London, UK.

13. Kernick DP. Introduction to health economics for the medical practitioner. Postgraduate medical journal. 2003;79(929):147-150.

14. O'Reilly D, Gaebel K, Xie F, Tarride JE, Goeree R. (2011). Health economic evaluations help inform payers of the best use of scarce health care resources. International journal of circumpolar health. 2011 ;70(4): 419-427.

15. Sutton AJ, Breheny K, Deeks J, Khunti K, Sharpe C, Ottridge RS, Stevens PE, Cockwell P, Kalra PA, Lamb EJ, eGFR-C study group. Methods Used in Economic Evaluations of Chronic Kidney Disease Testing-A Systematic Review. PloS one. 2015;10(10), p.e0140063.

16. Gray AM, Wilkinson T. Economic evaluation of healthcare interventions: old and new directions. Oxford Review of Economic Policy, 2016;32(1 ):102-121.

17. Centers for Disease Control and Prevention (CDC), Division for Heart Disease and Stroke Prevention (DHDSP). Glossary of Terms. (n.d.). http://www.cdc.gov/dhdsp/programs/spha/economic evaluation/docs/economic evaluation glossary. pdf

18. BMJ Clinical Evidence. (N.d.). http://clinicalevidence.bmi.com/x/set/static/ebm/toolbox/678253.html (Accessed April 29, 2016)

19. Botchkarev A. Toward Development of a New Health Economic Evaluation Definition. Journal of Economics Bibliography, 2016;3(4). In print.

20. Miller T R, Hendrie DV. Economic evaluation of public health laws and their enforcement. Public Health Law Research, February, 2012. http://ssrn.com/abstract=2012618

21. Hoomans T, Severens JL. Economic evaluation of implementation strategies in health care. Implementation Science. 2014; 9(1 ):1-6.

22. Bassi J, Lau F. Measuring value for money: a scoping review on economic evaluation of health information systems. Journal of American Medical Information Association. 2013;20:792-801.

23. Chhatwal J, He T. Economic evaluations with agent-based modelling: an introduction. Pharmacoeconomics. 2015;33(5):423-433.

24. Rochau U, Schwarzer R, Jahn B, Sroczynski G, Kluibenschaedl M, Wolf D, Radich J, Brixner D, Gast G, Siebert U. Systematic assessment of decision-analytic models for chronic myeloid leukemia. Applied health economics and health policy. 2014;12(2) :103-115.

25. Rogers PJ, Stevens K, Boymal J. Qualitative cost-benefit evaluation of complex, emergent programs. Evaluation and Program Planning. 2009;32(1):83-90. http://dx.doi.org/10.1016/i.evalprogplan.2008.08.005

26. Niessen LW, Bridges J, Lau BD, Wilson RF, Sharma R, Walker DG, Frick KD, Bass EB. Assessing the Impact of Economic Evidence on Policymakers in Health Care-A Systematic Review. AHRQ Publication No. 12(13)-EHC133-EF. 2012. http://www.ncbi.nlm.nih.gov/sites/books/NBK114636

27. Smith N, Mitton C, Peacock S. Qualitative methodologies in healthDcare priority setting research. Health economics. 2009;18(10):1163-1175. DOI: 10.1002/hec.1419

28. Brousselle A, Lessard C. Economic evaluation to inform health care decision-making: promise, pitfalls and a proposal for an alternative path. Social science & medicine. 2011 ;72(6):832-839.

29. Mihaylova B, Briggs A, O'Hagan A, Thompson SG. Review of statistical methods for analysing healthcare resources and costs. Health economics. 2011;20(8):897-916.

30. DeFauw MC. Cost Effectiveness Analysis in Healthcare Decision-Making: Stochastic Modeling and Statistical Inference (Doctoral dissertation, The University of Michigan). 2011. http://deepblue.lib.umich.edu/bitstream/handle/2027.42/89730/mcdefauw 1 .pdf?sequence=1

31. David MC, Bensink M, Higashi H, Donald M, Alati R, Ware RS. Monte Carlo simulation of the costeffectiveness of sample size maintenance programs revealed the need to consider substitution sampling. Journal of clinical epidemiology. 2012;65(11):1200-1211.

32. O'Hagan A, Stevens JW. Bayesian methods for design and analysis of cost-effectiveness trials in the evaluation of health care technologies. Statistical Methods in Medical Research. 2002;11(6):469-490.

33. Cooper NJ, Sutton AJ, Abrams KR, Turner D, Wailoo A. Comprehensive decision analytical modelling in economic evaluation: a Bayesian approach. Health Economics. 2004;13(3):203-226.

34. Revill P, Ochalek J, Lomas J, Nakamura R, Woods B, Rollinger A, Suhrcke M, Sculpher M, Claxton K. Cost-effectiveness thresholds: guiding health care spending for population health improvement. 2015. http://www.idsihealth.org/wp-content/uploads/2015/01/CE-Thresholds-iDSI-Working-Group-FinalReport.pdf

35. Torrance GW. Preferences for health outcomes and cost-utility analysis. The American journal of managed care. 1997;3(Suppl 1):S8-S20.

36. Luyten J, Naci H, Knapp M. Economic evaluation of mental health interventions: an introduction to cost-utility analysis. Evidence Based Mental Health. 2016. doi:10.1136/eb-2016-102354

37. Abdelhamid A, Shemilt I. Glossary of terms. Evidence-Based Decisions and Economics: Health Care, Social Welfare, Education and Criminal Justice, Second Edition, 2011. pp.186-197.

38. Rabarison KM, Bish CL, Massoudi MS, Giles WH. Economic evaluation enhances public health decision making. Frontiers in public health 3. 2015. http://dx.doi.org/10.3389/fpubh.2015.00164

39. Zarnke KB, Levine MA, O'Brien BJ. Cost-benefit analyses in the health-care literature: don't judge a study by its label. Journal of clinical epidemiology. 1997;50(7):813-822.

40. National Institute for Health and Clinical Excellence (NICE). Guide to the Single Technology Appraisal Process. NICE. 2009. https://www.nice.org.uk/Media/Default/About/what-we-do/NICEguidance/NICE-technology-appraisals/Guide-to-the-single-technology-appraisal-process.pdf (Accessed 20th April 2016).

41. Simoens S. Factors affecting the cost effectiveness of antibiotics. Chemotherapy research and practice. 2011. doi:10.1155/2011/249867