What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review

Abstract

Discrete choice experiments (DCEs) are a way to assess priority-setting in health care provision. This approach allows for the evaluation of individuals’ preferences as a means of adding criteria to traditional quality-adjusted life year analysis. The aim of this systematic literature review was to identify attributes for designing a DCE in order to then develop and validate a framework that supports decision-making on health technologies. Our systematic literature review replicated the methods and search terms used by de Bekker-Grob et al. 2012 and Clark et al. 2014. The Medline database was searched for articles dated between 2008 and 2015. The search was limited to studies in English that reflected general preferences and were choice-based, published as full-text articles and related to health technologies. This study included 72 papers, 52% of which focused on DCEs on drug treatments. The average number of attributes used in all included DCE studies was 5.74 (SD 1.98). The most frequently used attributes in these DCEs were improvements in health (78%), side effects (57%) and cost of treatment (53%). Other, less frequently used attributes included waiting time for treatment or duration of treatment (25%), severity of disease (7%) and value for money (4%). The attributes identified might inform future DCE surveys designed to study societal preferences regarding health technologies in order to better inform decisions in health technology assessment.

About the Authors:

Marta Trapero-Bertran

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Writing – original draft, Writing – review & editing

Affiliations Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain, Research Institute for Evaluation and Public Policies (IRAPP), Universitat Internacional de Catalunya (UIC), Barcelona, Spain

Beatriz Rodríguez-Martín

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Writing – original draft, Writing – review & editing

* E-mail: [email protected]

Affiliations Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain, Faculty of Health Sciences, University College Dublin, Dublin, Ireland

ORCID logo http://orcid.org/0000-0003-3939-0299

Julio López-Bastida

Roles Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Project administration, Supervision, Writing – original draft, Writing – review & editing

Affiliation: Department of Nursing, Physiotherapy and Occupational Therapy, University of Castilla La-Mancha (UCLM), Talavera de la Reina (Toledo), Spain

Introduction

In health care systems around the world, decision-makers are faced with competing demands and insufficient resources, even in the wealthiest countries. In these circumstances, it is not possible to provide all available and potentially beneficial health care to those who could benefit from it, and priority-setting is therefore needed. Policy-makers should take into account the views of the general population in setting health priorities, as is done in the United Kingdom [1]. Public involvement in health care decision-making should be a policy objective, although there is an absence of empirical evidence on how society might value different health interventions [2].

There is substantial literature on the different methods available to engage the public in health care decision-making [3–5]. As noted by Whitty et al. [6], Ryan and colleagues provide a comprehensive systematic review and comparative assessment of the methods that can be used to elicit public preferences for health care [7], concluding that “there is no single, best method to gain public opinion”. Nevertheless, they do make recommendations regarding the appropriateness of selected qualitative and quantitative techniques. Two of their preferred methods, the citizens’ jury and discrete choice experiments (DCEs), have been gaining prominence in the health literature in recent years [6, 8–12]. Each is associated with a number of features that make them particularly attractive for public engagement [6], rendering them worthy of further consideration. The DCE approach was chosen for this systematic literature review because it was aimed at informing a pilot study using a discrete choice experiment to quantify individual preferences regarding the use of public funding for orphan drugs.

The DCE has become a useful instrument for quantifying preferences related to health care priority-setting [6, 8, 13–16]. It has been used to (1) measure preferences regarding a health care service, (2) measure preferences regarding the distribution of health care within a population and (3) to assess preferences for the funding of health care [17–19]. Although use of the method by policy-makers has not yet become widespread, it has been applied to elicit social preferences for health care funded with public money [20].

A DCE survey can be administered relatively easily to a large, randomly selected representative sample of the population [7]. It is arguably a less resource-intensive method of community engagement than many other approaches, although resources and costs would likely be high for large sample sizes. A DCE measures not only the direction of preferences around a topic (e.g., Should health gains attributed to young children be weighted more heavily than those attributed to older people?), but also the relative strength of preferences for one policy choice alternative compared with another (e.g., How much extra weight should be attributed to young children?), as well as the trade-offs that respondents would be willing to make between different characteristics of that choice. The usefulness of most preference-based approaches (including DCEs) may be limited when the respondents represent what might be called a naïve sample of the general public–that is, one comprising individuals who lack personal knowledge or experience on the issue, and thus little weight can be given to the results [7, 21].

A DCE provides a different way–compared with other approaches, such as small‐scale discussions or focus groups–to assess priority-setting based on the valuation of some attributes. DCEs are based on the assumption that health care interventions, services or policies can be described by their attributes and that an individual’s valuation depends upon the levels of these attributes. In a DCE, respondents are asked to choose between two or more alternatives. The resulting choices reveal an underlying utility function. For example, the DCE approach allows for the evaluation of individuals’ preferences for adding criteria to traditional quality-adjusted life year (QALY) analysis. The DCE approach also facilitates greater knowledge of the relative importance of the various attributes and the trade-offs that individuals are willing to make between these attributes.

This type of research and its applications are crucial for identifying the current impact of new health technologies on health and economics and, therefore, for assessing their effectiveness. DCEs also afford an opportunity to assess societal preferences. This type of research could serve as the basis for an integrated and harmonized approach to assessing public policies on new health technologies in the European Union.

De Bekker Grob et al. [8] published a recent review on preferences of consumers, patients and health professionals for all types of health care resources. They focused on the experimental design of DCEs, estimation procedures, the validity of responses and the definition of the attributes and their respective levels that should be used for DCEs on health technologies. The attributes found in their review were monetary measure, time, risk, health status domain, health care, and other. No further description of the attributes was given, so there were no well-defined inputs to be used to design a DCE for a particular context. Clark et al. [22] published a more recent DCE review. This paper updated the paper by de Bekker Grob et al. [8] and explored trends in DCEs used in health economics. It concluded that the use of DCEs in health care continues to grow dramatically across a broad range of countries. Thus, DCE results may be influencing decisions in a wider range of geographical settings. Little description and detail regarding attributes and their respective levels were provided for inclusion in future DCE exercises. There have been several literature reviews of DCEs in health care in general (such as de Bekker Grob et al. and Clark et al.) [8, 22], but not of DCEs in health technology assessment (HTA). Decisions regarding HTA also involve public resources, however, and it is therefore important to establish approaches for prioritizing health technology resources. Accordingly, determining the attributes that should be considered in DCEs to inform HTA decisions should be a current research concern.

Materials and methods

This systematic literature review was conducted using the search terms and methods used in two recent published systematic reviews on DCEs covering the periods 2001–2008 [8] and 2009–2015 [22]. These methods involved the use of the Medline Ovid database to identify DCE studies on health care or health economics. These studies used the same search terms used by Ryan and Gerard [23], reflecting the different terms applied to refer to DCEs. The search terms included were “discrete choice experiment(s)”, “discrete choice model(l)ing”, “stated preference”, “part-worth utilities”, “functional measurement”, “paired comparisons”, “pairwise choices”, “conjoint analysis”, “conjoint measurement”, “conjoint studies” and “conjoint choice experiment(s)”.

In this study, the same database used in de Bekker Grob et al. [8] and Clark et al. [22] was used to search for articles published from January 2008 to December 2015. The same key words were also used. Papers in English and Spanish were retrieved, although the search terms used were in English only. Any paper explaining a DCE on health technologies was included. Review papers were excluded from the analysis but kept for the discussion section of this paper. De Bekker-Grob et al. [8] and Clark et al. [22] included studies that were choice-based and published as full-text articles and that applied to health care or health economics in general. Our review focused on health technologies and thus had a more limited scope. The search was extensive with respect to health care and health economics in general, but only papers related to health technologies were included in our systematic review. The objective was to evaluate DCEs on health technologies that reflected the preferences of patients, policy-makers, providers and the general public. Papers were excluded if they had the following characteristics: (a) they were duplicates; (b) they were not choice- or preference-based or they merely provided measurements but no descriptions of attributes; (c) neither the full text nor an abstract was found; (d) they did not apply to health technologies or to rural areas of developing countries; and (e) they did not involve human respondents. Grey literature was also searched using Google Scholar, although unfortunately no results were found. Each abstract and paper selected was carefully peer-reviewed, and data extraction was systematically and independently performed by two researchers. Whenever there was a discrepancy, papers were reviewed a second time to reach a consensus. Excel was used to summarize the results of this systematic literature review. A data extraction form included questions on the following: background (e.g., quartile of impact factor); sampling and sample characteristics (e.g., illness of respondents); general design of the DCE (e.g., number of attributes and description of attributes and levels covered); experimental design (e.g., method for creating choice sets); design validity (e.g., estimation procedure, model); and qualitative methods for enhancing the DCE process and results (e.g., pretesting of the DCE questionnaire). However, only the general design and experimental design features were presented in the results and discussion sections. Considerations relating to design validity and qualitative methods used to enhance the DCE process were beyond the scope of this paper.

The specific details of the template were dynamically adjusted during the piloting process, which included the revision of a few papers. Data were extracted in free-text form with no limitations on the number of free-text fields and as little categorization of data as possible to avoid the loss of detailed information. Descriptive analysis was undertaken to describe the most common attributes used and their corresponding levels. The attributes and levels for the DCE questionnaire on HTA are presented in a summary table. The table shows the attributes found in this literature review and the attributes identified in the previously published systematic literature reviews [8, 22].

To assess the methodological quality of the systematic literature review, the PRISMA checklist was used [24]. In addition, a DCE quality assessment tool [25] was used to assess the validity of the studies and their attributes and levels.

This approach was devised following the guidance of Mandeville et al. [25], who covered all four key stages of a DCE (choice task design; experimental design; conduct; and analysis) using a list of 13 criteria drawn from an earlier study [26]. The authors assessed whether each criterion for each study was met. If the criterion was met, it was indicated with a green colour. If there was insufficient information to judge whether a criterion was met, then an orange colour was used. A red colour indicated that the criterion was not met. This type of qualitative analysis is important for validating the results from this systematic literature review.

Results

Fig 1 shows the flowchart for the identification of studies, with reasons for exclusion.

[Figure omitted. See PDF.]

Fig 1. Number of hits identified by the literature review.

https://doi.org/10.1371/journal.pone.0219905.g001

Overall, the search strategy identified 384 titles (after duplicates were excluded) from a pool of studies with the potential for inclusion in this review. Based on the abstracts, 160 papers were ordered and manually reviewed. Of these 160 articles, 72 were included in this study [14, 27–96]. S1 Table, included in the supplemental material, provides details of the PRISMA checklist used to assess the methodological quality of the systematic literature reviews.

The number of studies published on DCEs on general preferences regarding health technologies over time is as follows: 18 articles (25%) were published before 2010, 30 articles (41.6%) appeared between 2010 and 2011, and the remaining 21 articles (29%) were published between 2012 and 2013. The years with the greatest research output were 2011 and 2012 (16 articles and 15 articles published, respectively). The average sample used across the 72 studies included 299 individuals with a mean age of 59.6 years; an average of 44.88% of the respondents were female. In seven studies, only the age interval was reported; in those cases, the average of the age interval was taken. The largest number of the DCEs identified were conducted in the Netherlands, although significant numbers were also conducted in the United Kingdom, Canada, Germany and the United States. The 72 papers covered 30 different diseases, such as chronic obstructive pulmonary disease (COPD), depression and hepatitis B. The most studied preferences related to cancer (26%), followed by attention deficit disorder (4%) and osteoporosis (4%). Only one paper [61] was found that examined preferences relating to orphan drugs for rare diseases. S2 Table, included in the supplemental material, provides more detailed information about the health technologies, the attributes and the levels for each paper.

Fig 2 presents the validity assessment for all included studies. Overall, while the choice task design and the experimental design of the studies were more robust than expected, there were significant weaknesses with regard to conduct and analysis of the studies. In terms of choice task, attributes and levels were identified through qualitative work with the target population in 24.8% of the studies. In 16% of the studies, there was no opt-out or status quo option, nor any justification of a forced choice for the attributes selected. In terms of the experimental design, 27% of papers did not have a design that was optimal or statistically efficient. However, most of the relevant problems with the validity of the papers pertained to the pilot testing conducted among the target population and the lack of a pooled analysis from different subgroups: in 42% of the papers the authors did not conduct a pilot test to inform the design of the questionnaire, and 63% did not include any pooled analysis from different population subgroups. In an assessment of the validity of experimental design and analytic approach, it is necessary to examine current practices in DCEs.

[Figure omitted. See PDF.]

Fig 2. Validity assessment of included studies.

https://doi.org/10.1371/journal.pone.0219905.g002

Most of the studies (53%) assessed societal preferences regarding pharmacological treatments; the rest focused on medical devices. Of the latter, 17% related to diagnostic technologies and 26% to therapeutic technologies (specifically, 13% were on assistive technology devices used directly by patients, 5% were on medical devices to assist medical professionals, and 4% were on artificial body parts implanted through medical procedures). The average number of attributes was 5.74 (SD 1.98), with a minimum value of 2 and a maximum value of 12. Each attribute had an average of 3.26 levels (SD 1.11), with a minimum value of 1 and a maximum value of 18. The six most common attributes used in the DCEs (n = 72) were (a) improvement in health (78%), (b) side effects (57%), (c) costs (53%), (d) waiting time for treatment or duration of treatment (25%), (e) severity of disease (7%), and (f) value for money (4%). When the focus was only on papers that assessed preferences in relation to drugs (n = 36), the relative importance of the attributes remained the same: improvement in health (55.56%), costs (50%), adverse events (41.67%), and mode of administration (22.22%), followed by discomfort and pain (16.67%), treatment duration (19.44%) and waiting time (2.78%). These attributes reflected the general preferences of several groups, including patients, the general public patients and other stakeholders. The majority of the papers (n = 60) referred to patients’ preferences. In terms of attributes revealing preferences related to efficiency, availability of other treatments and value for money were considered relevant attributes [14]. Availability of other treatments refers to the existence of alternative treatments for the same disease. Value for money refers to how efficiently resources are used (e.g., doctor time, hospital beds, drugs) in the national health system and is based on the relationship between the cost of treatment and the health benefits it provides. Hence, although these two attributes were not the most commonly used, they were also considered for inclusion in the DCE survey.

In terms of levels, the papers reviewed most commonly referred to the following levels of administration: oral, subcutaneous, intravenous or injection. These papers also referred to the following levels of pain or discomfort: none, mild, moderate or severe. Therefore, the terms mild, moderate, and severe were adopted to describe the levels for as many attributes as possible. See Table 1 for details on the most used attributes and their respective levels. This table also includes the attributes and levels described in the two previously published systematic reviews [8, 22]. Both studies highlight the monetary measure and the time- and health care-related attributes as the most frequent ones to be considered in a DCE.

[Figure omitted. See PDF.]

Table 1. Attributes and levels for use in a DCE questionnaire on HTA.

https://doi.org/10.1371/journal.pone.0219905.t001

Only 10 papers (13.8%) offered partial access to the questionnaire used to carry out the DCE and the specific questions formulated for respondents. The rest of the papers did not offer access to the survey. The most common means of administering the survey was a self-report questionnaire (35.29%); questionnaires were administered through an interviewer in 7.35% of cases or a computer in 8.82%. The rest of the papers did not report the mode of survey administration.

Regarding experimental design, 43% (n = 31) of the studies had a fractional factorial design, whereas 13% (n = 10) had a full factorial design. The remaining 27 studies did not report the type of DCE design. Thirty-one studies measured main effects only, whereas seven studies reported main effects with 2-way interactions. In 32% of the DCEs reviewed, the effects evaluated were not reported. More than half of the papers (55%) used orthogonal arrays to create choice sets, while 14% used D-efficiency methods. Five papers combined two methods: either orthogonal arrays with D-efficiency or other methods. Three papers used other methods to create choice sets, and 18 did not explain the method used. In terms of the estimation procedure, the multinomial logit was the most common model used to analyse DCE preferences (22%), followed by random effects logit (13%) and logit (12%).

Discussion

This study fills a gap in systematic reviews of the literature aimed at identifying the most relevant attributes and levels for measuring public preferences regarding health technologies by means of a DCE. Six attributes were identified from de Bekker Grob et al. (2012) [8] and Clark et al. (2014) [22]: health status domain, monetary measure, time, risk, health care and other, some of which were too subjective to build questions for a DCE survey. No levels were defined or detailed in these papers and no additional information was given concerning the definitions of these attributes. In contrast, our literature review found the following attributes: improvement in health, side effects, cost (price) of treatment, waiting time for treatment or treatment duration, severity of disease and value for money. The attributes included in the previous systematic reviews are too wide and general to understand. No definitions were provided by the authors, so it was difficult to evaluate the complementarity between the results of the two systematic literature reviews, even though they used the same literature review methods. It will be important for future research to describe these attributes and their respective levels in as detailed a manner as possible, so that they can be applied with no uncertainties regarding what is encompassed in their definition. In addition, a complete description will be helpful in providing information for the design of future DCEs on HTA. Because public preferences might change greatly over time, depending on current situations worldwide, it was decided to incorporate papers published between 2008 and 2015 –i.e., a period of 7 years. Regarding the optimum number of attributes to include in a DCE, Marshal et al. [97] identified and described recent applications of conjoint analysis to determine what combination of a limited number of attributes was most influential on respondent choice or decision-making. In their review, they found that most surveys included 6 attributes, with the number ranging from 3 to 16. Therefore, it seems that a larger number of attributes should be used to better capture the criteria on which people base their preferences related to health care. However, many attributes make the decision task more difficult and hence render the outcomes less reliable. The number of attributes found in this systematic review–six–seems a sensible and adequate number to be potentially included in a DCE.

Orphan drugs are unlikely to be efficient (provide value for money) due to the high price paid for often modest effectiveness. It is important to identify all appropriate criteria that will help in the “correct” evaluation of the potential impact and benefit generated in society. Unfortunately, only one paper [61] was found that studied preferences relating to orphan drugs for rare diseases. The authors investigated public preferences regarding public funding for orphan drugs used to treat both rare and common diseases, using a convenience sample of university students. They found that when all other variables were held constant, the respondents did not prefer to have the government spend more for orphan drugs used to treat rare diseases and that they weighted the relevant attributes of coverage decisions similarly for both rare and common diseases. More DCEs on orphan drugs should be conducted to generate more evidence on the particular attributes and levels for this kind of drug.

The inclusion of either cost or improvement in health and value for money as attributes helps to capture the preferences of respondents, although it could lead to double-counting. None of the papers found in this systematic review included either combination of those attributes; however, it is important to be aware of the potential for double-counting that can occur as a result of the inclusion of such similar attributes.

Although there were significant weaknesses in terms of the validity assessment of the included studies, important and essential issues–such as no overlap between the attributes, the use of unidimensional attributes in the questionnaires, the use of the correct target population and the appropriate use of an econometric model for the choice task design–were common characteristics for most of the studies. Hence, despite some weaknesses regarding validity, the most important criteria for these types of studies were included overall.

DCEs have been previously used in other published studies to gain insight into the criteria that were important for decision-makers in health care priority-setting [14, 40, 52, 64, 65]. These five papers were included in this literature review. The rest of the papers (n = 67) focused on patient preferences. The type of attributes used in the papers that focused on policy-makers’ opinions were quite different from those that sought to identify patients’ preferences. For instance, six intervention-related attributes were included in a paper [73] that measured the preferences of policy-makers and other health professionals, including disease severity, budget impact, cost-effectiveness (incremental cost per QALY and number of QALYs gained per patient), uncertainty regarding the probability of doubling costs per QALY, national savings in costs related to absence from work per year and the composition of the health gain. Disease severity and cost of treatment are included in both literature reviews, as is health improvement; however, questions related to cost-effectiveness or national savings when the target audience includes patients are more difficult to ask. For that reason, availability of other treatments and value for money were included as relevant attributes. Attributes might differ, depending on the survey target audience. In this case, all audiences have been included.

Subsequent research is needed to further develop DCE attributes and levels for various specific technologies and diseases. One possible approach might be to investigate in a more in-depth manner the methods that led to the selection and identification of attributes in DCE studies (e.g., focus groups, interviews, literature, expert opinion), which could then be informative for future DCEs. Another approach might be to conduct DCEs for different diseases among different types of audiences to assess and validate attributes and thus help to inform future priority-setting decisions.

Conclusions

This systematic literature review was performed to identify the attributes that may better help decision-makers and patients to identify the criteria leading to decisions about health technologies. This study revealed that attributes such as improvements in health, treatment side effects, treatment cost (price), waiting time for treatment or treatment duration, severity of disease and value for money can be considered to better capture and describe societal preferences in relation to HTA. This topic is of interest for preference practitioners, as it can help them, first, to build the best survey on health technology and, then, to aid public decision-makers in identifying the treatments that should be implemented or funded, in accordance with the population’s preferences.

Supporting information

[Figure omitted. See PDF.]

S1 Table. PRISMA 2009 checklist.

https://doi.org/10.1371/journal.pone.0219905.s001

(DOC)

S2 Table. Description of attributes and levels by device or HTA.

https://doi.org/10.1371/journal.pone.0219905.s002

(DOCX)

Citation: Trapero-Bertran M, Rodríguez-Martín B, López-Bastida J (2019) What attributes should be included in a discrete choice experiment related to health technologies? A systematic literature review. PLoS ONE 14(7): e0219905. https://doi.org/10.1371/journal.pone.0219905

References

1. Department of Health. Involving patients and the public in healthcare: a discussion document. London: Practitioners RCOG; 2001.

2. Dolan P, Shaw R, Tsuchiya A, Williams A. QALY maximisation and people’s preferences: A methodological review of the literature. Heal Econ 2005;14:197–208.

3. Mitton C, Smith N, Peacock S, Evoy B, Abelson J. Public participation in health care priority setting: A scoping review. Health Policy 2009; 91:219–28. pmid:19261347

4. Rowe G, Frewer LJ. A typology of public engagement mechanisms. Science, Technology & Human Values 2005; 30: 251–90.

5. Mullen PM. Public involvement in health care priority setting: an overview of methods for eliciting values. Health Expect 1999; 2:222–34. pmid:11281899

6. Whitty JA, Ratcliffe J, Chen G, Scuffham PA. Australian Public Preferences for the Funding of New Health Technologies: A Comparison of Discrete Choice and Profile Case Best Worst Scaling Methods. Med Decis Making 2014; 34:638–54. pmid:24713695

7. Ryan M, Scott DA, Reeves C, Bate A, van Teijlingen ER, Russell EM, et al. Eliciting public preferences for healthcare: a systematic review of techniques. Health Technol Assess 2001; 5:1–186

8. de Bekker-Grob EW, Ryan M, Gerard K. Discrete choice experiments in health economics: a review of the literature. Heal Econ 2012;21:145–72.

9. Menon D, Stafinski T. Engaging the public in priority-setting for health technology assessment: findings from a citizens’ jury. Health Expect 2008; 11: 282–93. pmid:18816324

10. Mooney G. A Handbook on Citizens’ Juries with Particular Reference to Health Care [internet]. 2010

11. Coote A, Lenaghan J. Citizens’ Juries: Theory into Practice. London: Institute for Public Policy Research; 1997.

12. Moretto N, Kendall E, Whitty J, Byrnes J, Hills AP, Gordon L, et al. Yes, the government should tax soft drinks: findings from a citizens’ jury in Australia. Int J Environ Res Public Health 2014; 11: 2456–71. pmid:24583829

13. Ratcliffe J. Public preferences for the allocation of donor liver grafts for transplantation. Health Econ 2000; 9: 137–48. pmid:10721015

14. Green C, Gerard K. Exploring the social value of health-care interventions: a stated preference discrete choice experiment. Heal Econ 2009;18:951–76.

15. Whitty JA, Scuffham PA, Rundle-Thiele SR. Public and decision maker stated preferences for pharmaceutical subsidy decisions: a pilot study. Appl Health Econ Health Policy 2011; 9: 73–9. pmid:21332251

16. Whitty JA, Lancsar E, Rixon K, Golenko X, Ratcliffe J. A systematic review of stated preference studies reporting public preferences for healthcare priority setting. Patient 2014; 7:365–86. pmid:24872225

17. Schwappach DLB. Does it matter who you are or what you gain? An experimental study of preferences for resource allocation. Health Econ 2003; 12: 255–67. pmid:12652513

18. Scuffham PA, Whitty JA, Taylor M, Saxby R. Health system choice: A pilot discrete choice experiment eliciting the preferences of British and Australian citizens. Appl Health Econ Health Policy 2010; 8: 89–97. pmid:20175588

19. Gyrd-Hansen D, Slothuus U. The citizen’s preferences for financing public health care: a Danish survey. Int J Health Care Finance Econ 2002; 2: 25–36. pmid:14625906

20. Lancsar E, Wildman J, Donaldson C, Ryan M, Baker R. Deriving distributional weights for QALYs through discrete choice experiments. J Health Econ 2011; 30: 466–78. pmid:21310500

21. Ryan M, Gerard K, Amaya-Amaya M. Using Discrete Choice Experiments to Value Health and Health Care. Dordrecht, Netherlands: Springer; 2008.

22. Clark MD, Determann D, Petrou S, Moro D, de Bekker-Grob EW. Discrete choice experiments in health economics: a review of the literature. Pharmacoeconomics 2014;32:883–902. pmid:25005924

23. Ryan M, Gerard K. Using discrete choice experiments to value health care programmes: current practice and future research reflections. Appl Heal Econ Heal Policy 2003;2:55–64.

24. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions explanation and elaboration. BMJ 2009: 339.

25. Mandeville K, Lagarde M, Hanson K. The use of discrete choice experiments to inform health workforce policy: a systematic review. BMC Health Serv Res 2014;14:367. pmid:25179422

26. Lancsar E, Louviere J. Conducting discrete choice experiments to inform healthcare decision making: A user’s guide. Pharmacoeconomics 2008;26:661–77. pmid:18620460

27. Ahmed SF, Smith WA, Blamires C. Facilitating and understanding the family’s choice of injection device for growth hormone therapy by using conjoint analysis. Arch Dis Child 2008;93:110–4. pmid:17412745

28. Augustovski F, Beratarrechea A, Irazola V, Rubinstein F, Tesolin P, Gonzalez J, et al. Patient preferences for biologic agents in rheumatoid arthritis: A discrete-choice experiment. Value Heal 2013;16:385–93.

29. Bøgelund M, Vilsbøll T, Faber J, Henriksen JE, Gjesing RP, Lammert M. Patient preferences for diabetes management among people with type 2 diabetes in Denmark—a discrete choice experiment. Curr Med Res Opin 2011;27:2175–83. pmid:21981417

30. Bonnichsen O. Elicitation of ostomy pouch preferences: A discrete-choice experiment. Patient 2011;4:163–75. pmid:21766912

31. Bridges JFP, Kinter ET, Schmeding A, Rudolph I, Mühlbacher A. Can patients diagnosed with schizophrenia complete choice-based conjoint analysis tasks? Patient 2011;4:267–75. pmid:21995832

32. Bridges JFP, Lataille AT, Buttorff C, White S, Niparko JK. Consumer preferences for hearing aid attributes: a comparison of rating and conjoint analysis methods. Trends Amplif 2012;16:40–8. pmid:22514094

33. Bridges JFP, Mohamed AF, Finnern HW, Woehl A, Hauber AB. Patients’ preferences for treatment outcomes for advanced non-small cell lung cancer: A conjoint analysis. Lung Cancer 2012;77:224–31. pmid:22369719

34. Bridges JFP, Searle SC, Selck FW, Martinson NA. Designing family-centered male circumcision services: A conjoint analysis approach. Patient 2012;5:101–11. pmid:22510052

35. Bunge EM, de Bekker-Grob EW, van Biezen FC, Essink-Bot ML, de Koning HJ. Patients’ preferences for scoliosis brace treatment: a discrete choice experiment. Spine 2010;35:57–63. pmid:20023605

36. Burnett HF, Regier DA, Feldman BM, Miller FA, Ungar WJ. Parents’ preferences for drug treatments in juvenile idiopathic arthritis: A discrete choice experiment. Arthritis Care Res 2012;64:1382–91.

37. Chancellor J, Martin M, Liedgens H, Baker MG, Müller-Schwefe GHH. Stated preferences of physicians and chronic pain sufferers in the use of classic strong opioids. Value Heal 2012;15:106–17.

38. Damen THC, de Bekker-Grob EW, Mureau MAM, Menke-Pluijmers MB, Seynaeve C, Hofer SOP, et al. Patients’ preferences for breast reconstruction: a discrete choice experiment. J Plast Reconstr Aesthet Surg 2011;64:75–83. pmid:20570232

39. Darbà J, Restovic G, Kaskens L, Balbona MA, Carbonell A, Cavero P, et al. Patient preferences for osteoporosis in Spain: A discrete choice experiment. Osteoporos Int 2011;22:1947–54. pmid:20838770

40. de Bekker-Grob EW, Essink-Bot ML, Meerding WJ, Pols HAP, Koes BW, Steyerberg EW. Patients’ preferences for osteoporosis drug treatment: A discrete choice experiment. Osteoporos Int 2008;19:1029–37. pmid:18193329

41. de Bekker-Grob EW, Essink-Bot ML, Meerding WJ, Koes BW, Steyerberg EW. Preferences of GPs and patients for preventive osteoporosis drug treatment: A discrete-choice experiment. Pharmacoeconomics 2009;27:211–9. pmid:19354341

42. de Bekker-Grob EW, Hofman R, Donkers B, van Ballegooijen M, Helmerhorst TJM, Raat H, et al. Girls’ preferences for HPV vaccination: A discrete choice experiment. Vaccine 2010;28:6692–7. pmid:20708696

43. de Bekker-Grob EW, Hol L, Donkers B, van Dam L, Habbema JDF, van Leerdam ME, et al. Labeled versus unlabeled discrete choice experiments in health economics: An application to colorectal cancer screening. Value Heal 2010;13:315–23.

44. Deal K, Marshall D, Dabrowski D, Charter A, Bukoski M, Moayyedi P. Assessing the value of symptom relief for patients with gastroesophageal reflux disease treatment: Willingness to pay using a discrete choice experiment. Value Heal 2013;16:588–98.

45. Essers BAB, Dirksen CD, Prins MH, Neumann HAM. Assessing the public’s preference for surgical treatment of primary basal cell carcinoma: A discrete-choice experiment in the South of the Netherlands. Dermatologic Surg 2010;36:1950–5.

46. Essers BAB, Van Helvoort-Postulart D, Prins MH, Neumann M, Dirksen CD. Does the inclusion of a cost attribute result in different preferences for the surgical treatment of primary basal cell carcinoma?: A comparison of two discrete-choice experiments. Pharmacoeconomics 2010;28:507–20. pmid:20387912

47. Fegert JM, Slawik L, Wermelskirchen D, Nübling M, Mühlbacher A. Assessment of parents’ preferences for the treatment of school-age children with ADHD: a discrete choice experiment. Expert Rev. Pharmacoeconomics Outcomes Res 2011;11:245–52.

48. Grutters JPC, Kessels AGH, Dirksen CD, Van Helvoort-Postulart D, Anteunis LJC, Joore MA. Willingness to accept versus willingness to pay in a discrete choice experiment. Value Heal 2008;11:1110–9.

49. Guo N, Marra CA, FitzGerald JM, Elwood RK, Anis AH, Marra F. Patient preference for latent tuberculosis infection preventive treatment: a discrete choice experiment. Value Health 2011;14:937–43. pmid:21914516

50. Hauber AB, Arden NK, Mohamed AF, Johnson FR, Peloso PM, Watson DJ, et al. A discrete-choice experiment of United Kingdom patients’ willingness to risk adverse events for improved function and pain control in osteoarthritis. Osteoarthr Cartil 2013;21:289–97. pmid:23182815

51. Herbild L, Bech M, Gyrd-Hansen D. Estimating the Danish populations’ preferences for pharmacogenetic testing using a discrete choice experiment. the case of treating depression. Value Heal 2009;12:560–7.

52. Hill M, Fisher J, Chitty LS, Morris S. Women’s and health professionals’ preferences for prenatal tests for Down syndrome: a discrete choice experiment to contrast noninvasive prenatal diagnosis with current invasive tests. Genet Med 2012;14:905–13. pmid:22935718

53. Hinoul P, Goossens A, Roovers JP. Factors determining the adoption of innovative needle suspension techniques with mesh to treat urogenital prolapse: A conjoint analysis study. Eur J Obstet Gynecol Reprod Biol 2010;151:212–6. pmid:20427115

54. Hodgkins P, Swinburn P, Solomon D, Yen L, Dewilde S, Lloyd A. Patient preferences for first-line oral treatment for mild-to-moderate ulcerative colitis: A discrete-choice experiment. Patient 2012;5:33–44. pmid:22077619

55. Hol L, de Bekker-Grob EW, van Dam L, Donkers B, Kuipers EJ, Habbema JDF, et al. Preferences for colorectal cancer screening strategies: a discrete choice experiment. Br J Cancer 2010;102:972–80. pmid:20197766

56. Howard K, Salkeld G. Does attribute framing in discrete choice experiments influence willingness to pay? Results from a discrete choice experiment in screening for colorectal cancer. Value Heal 2009;12:354–63.

57. Howard K, Salkeld G, Pignone M, Hewett P, Cheung P, Olsen J, et al. Preferences for CT colonography and colonoscopy as diagnostic tests for colorectal cancer: A discrete choice experiment. Value Heal 2011;14:1146–52.

58. Ijzerman MJ, van Til JA, Bridges JFP. A comparison of analytic hierarchy process and conjoint analysis methods in assessing treatment alternatives for stroke rehabilitation. Patient 2012;5:45–56. pmid:22185216

59. Johnson FR, Lancsar E, Marshall D, Kilambi V, Mühlbacher A, Regier DA, et al. Quantifying women’s stated benefit-risk trade-off preferences for IBS treatment outcomes. Value Heal 2010;13:418–23.

60. Kinter ET, Prior TJ, Carswell CI, Bridges JF. A comparison of two experimental design approaches in applying conjoint analysis in patient-centered outcomes research: a randomized trial. Patient 2012;5:279–94. pmid:23145548

61. Kolasa K, Zwolinski KM, Kalo Z, Hermanowski T. Potential impact of the implementation of multiple-criteria decision analysis (MCDA) on the Polish pricing and reimbursement process of orphan drugs. Orphanet J Rare Dis 2016;11:23. pmid:26965710

62. Koopmanschap MA, Stolk EA, Koolman XHE. Dear policy maker: Have you made up your mind? A discrete choice experiment among policy makers and other health professionals. Int J Technol Assess Health Care 2010;26:198–204.

63. Kruijshaar ME, Essink-Bot M-L, Donkers B, Looman CWN, Siersema PD, Steyerberg EW. A labelled discrete choice experiment adds realism to the choices presented: preferences for surveillance tests for Barrett esophagus. BMC Med Res Methodol 2009;9:31. pmid:19454022

64. Lathia N, Isogai PK, Walker SE, De Angelis C, Cheung MC, Hoch JS, et al. Eliciting patients’ preferences for outpatient treatment of febrile neutropenia: a discrete choice experiment. Support Care Cancer 2013;21:245–51. pmid:22684150

65. Laver K, Ratcliffe J, George S, Burgess L, Crotty M. Is the Nintendo Wii Fit really acceptable to older people?: a discrete choice experiment. BMC Geriatr 2011;11:64. pmid:22011360

66. Lee WC, Joshi A V., Woolford S, Sumner M, Brown M, Hadker N, et al. Physicians’ preferences towards coagulation factor concentrates in the treatment of haemophilia with inhibitors: A discrete choice experiment. Haemophilia 2008;14:454–65. pmid:18282152

67. Manjunath R, Yang JC, Ettinger AB. Patients’ preferences for treatment outcomes of add-on antiepileptic drugs: A conjoint analysis. Epilepsy Behav 2012;24:474–9. pmid:22770879

68. Marshall DA, Johnson FR, Kulin NA, Özdemir S, Walshd JME, Marshall JK, et al. How do physician assessments of patient preferences for colorectal cancer screening tests differ from actual preferences? A comparison in Canada and the United States using a stated-choice survey. Health Econ 2009;18:1420–39. pmid:19191268

69. McTaggart-Cowan HM, Shi P, Fitzgerald JM, Anis A, Kopec JA, Bai TR, et al. An evaluation of patients’ willingness to trade symptom-free days for asthma-related treatment risks: a discrete choice experiment. J Asthma 2008;45:630–8. pmid:18951253

70. Mentzakis E, Stefanowska P, Hurley J. A discrete choice experiment investigating preferences for funding drugs used to treat orphan diseases: an exploratory study. Heal Econ Policy Law 2011;6:405–33.

71. Mohamed AF, Kilambi V, Luo MP, Iyer RG, Li-McLeod JM. Patient and parent preferences for immunoglobulin treatments: a conjoint analysis. J Med Econ 2012;15:1183–91. pmid:22846048

72. Mohamed AF, Johnson F, Hauber AB, Lescrauwaet B, Masterson A. Physicians’ stated trade-off preferences for chronic hepatitis B treatment outcomes in Germany, France, Spain, Turkey, and Italy. Eur J Gastroenterol Hepatol 2012;24:419–26. pmid:22330238

73. Morel T, Ayme S, Cassiman D, Simoens S, Morgan M, Vandebroek M. Quantifying benefit-risk preferences for new medicines in rare disease patients and caregivers. Orphanet J Rare Dis 2016;11:70. pmid:27225337

74. Mühlbacher AC, Rudolph I, Lincke H-J, Nübling M. Preferences for treatment of Attention-Deficit/Hyperactivity Disorder (ADHD): a discrete choice experiment. BMC Health Serv Res 2009;9:149–58. pmid:19678946

75. Opuni M, Bishai D, Gray GE, McIntyre JA, Martinson NA. Preferences for characteristics of antiretroviral therapy provision in Johannesburg, South Africa: Results of a conjoint analysis. AIDS Behav 2010;14:807–15. pmid:19533322

76. Palumbo A, De La Fuente P, Rodríguez M, Sánchez F, Martínez-Salazar J, Muñoz M, et al. Willingness to pay and conjoint analysis to determine women’s preferences for ovarian stimulating hormones in the treatment of infertility in Spain. Hum Reprod 2011;26:1790–8. pmid:21558333

77. Pereira CCA, Mulligan M, Bridges JFP, Bishai D. Determinants of influenza vaccine purchasing decision in the US: A conjoint analysis. Vaccine 2011;29:1443–7. pmid:21185424

78. Petrou S, McIntosh E. Women’s preferences for attributes of first-trimester miscarriage management: A stated preference discrete-choice experiment. Value Heal 2009;12:551–9.

79. Pieterse AH, Stiggelbout AM, Marijnen CAM. Methodologic evaluation of adaptive conjoint analysis to assess patient preferences: An application in oncology. Heal Expect. 2010;13:392–405.

80. Pignone MP, Brenner AT, Hawley S, Sheridan SL, Lewis CL, Jonas DE, et al. Conjoint analysis versus rating and ranking for values elicitation and clarification in colorectal cancer screening. J Gen Intern Med 2012;27:45–50. pmid:21870192

81. Pisa G, Freytag S, Schandry R. Chronic Obstructive Pulmonary Disease (COPD) Patients’ disease-related preferences: a study using conjoint analysis. Patient 2013;6:93–101. pmid:23526399

82. Regier DA, Diorio C, Ethier MC, Alli A, Alexander S, Boydell KM, et al. Discrete Choice Experiment to evaluate factors that influence preferences for antibiotic prophylaxis in pediatric oncology. PLoS One 2012;7.

83. Ryan M, Watson V. Comparing welfare estimates from payment card contingent valuation and discrete choice experiments. Health Econ 2009;18:389–401. pmid:18677721

84. Ryan M, Watson V, Entwistle V. Rationalising the “irrational”: A think aloud study of discrete choice experiment responses. Health Econ 2009;18:321–36. pmid:18651601

85. Sadique MZ, Devlin N, Edmunds WJ, Parkin D. The Effect of perceived risks on the demand for vaccination: results from a discrete choice experiment. PLoS One. 2013;8.

86. Sassi F, McKee M. Do clinicians always maximize patient outcomes? A conjoint analysis of preferences for carotid artery testing. J Health Serv Res Policy 2008;13:61–6. pmid:18416909

87. Shafey M, Lupichuk SM, Do T, Owen C, Stewart D. Preferences of patients and physicians concerning treatment options for relapsed follicular lymphoma: a discrete choice experiment. Bone Marrow Transplant. 2011;46:962–9. pmid:20935681

88. Skjoldborg US, Lauridsen J, Junker P. Reliability of the discrete choice experiment at the input and output level in patients with rheumatoid arthritis. Value Heal 2009;12:153–8.

89. Stockwell M, Rosenthal S, Sturm L, Mays R, Bair R, Zimet G. The effects of vaccine characteristics on adult women’s attitudes about vaccination: A conjoint analysis study. Vaccine 2011;29:4507–11. pmid:21527303

90. Sussex J, Rollet P, Garau M, Schmitt C, Kent A, Hutchings A. A pilot study of multicriteria decision analysis for valuing orphan medicines. Value Heal 2013;16:1163–9.

91. Sweeting KR, Whitty JA, Scuffham PA, Yelland MJ. Patient preferences for treatment of achilles tendon pain: Results from a discrete-choice experiment. Patient 2011;4:45–54. pmid:21766893

92. Torbica A, Fattore G. Understanding the impact of economic evidence on clinical decision making: a discrete choice experiment in cardiology. Soc Sci Med 2010;70:1536–43. pmid:20207466

93. van Dam L, Hol L, de Bekker-Grob EW, Steyerberg EW, Kuipers EJ, Habbema JDF, et al. What determines individuals’ preferences for colorectal cancer screening programmes? A discrete choice experiment. Eur J Cancer 2010;46:150–9. pmid:19683432

94. van Til J, Stiggelbout A, Ijzerman M. The effect of information on preferences stated in a choice-based conjoint analysis. Patient Educ Couns 2009;74:264–71. pmid:18954956

95. Waschbusch D, Cunningham C, Pelham W, Rimas H, Greiner A, Gnagy E, et al. A discrete choice conjoint experiment to evaluate parent preferences for treatment of young, medication naive children with ADHD. J Clin Child Adolesc Psychol 2011;40:546–61. pmid:21722027

96. Golan O, Hansen P, Kaplan G, Tal O. Health technology prioritizing new technologies and what are their relative weights? Health 2011;102(2–3):126–35.

97. Marshall D, Bridges JFP, Hauber B, Cameron R, Donnalley L, Fryie K, et al. Conjoint Analysis Application in Health- How are studies being designed and reported? Patient-Centered Outcomes Res 2010;3: 249–56.