Full text

Turn on search term navigation

Introduction

Given the sizeable burden and costs associated with mental disorders,1–6 the European Union has called for a systematic approach toward research in this area. The "ROAdmap for MEntal health Research and well-being in Europe" (ROAMER) project, funded by the European Commission's Seventh Framework Programme, was given the task of developing a mental health research roadmap based on expert consultation covering various domains in the field (ie, psychological research, biomedical research, research funding, infrastructures, and capacity building, research on social and economic aspects, and research on well-being and public health), as well as on the views of other stakeholders, such as service users, carers, professionals, and policy and funding institutions.7 Several work packages within ROAMER have already published about their priorities, such as public health research needs for improvement of mental health in Europe,8 the core aspects of biomedical and psychological mental health research and their possible role in diagnosis and treatment of mental disorders,9–13 as well as the views of national associations/organizations of psychiatrists, other mental health professionals, users and/or carers, and psychiatric trainees in the 27 countries of the European Union about the priorities for mental health research in Europe.14 Subsequently, within ROAMER, clinical mental health research was recognized as an area of particular interest, given the growing gap between the enormous output of neuroscience research and the actual impact of research on clinical treatment for patients with mental disorders.15 Clinical mental health research aims to translate such research findings by applied research aimed at improving the health of individuals with mental disorders and thus is required to bridge this gap. Current treatment of mental disorders usually requires complex and multilevel interventions targeting morbidity and mortality reduction, ie, prevention of suicide,16,17 detection and treatment of mental disorders by consultation models,18,19 and collaborative care models integrating primary care and specialty mental health care,20 and by integrating psychopharmacological, medical, psychotherapeutic, psychological, system, and societal interventions aimed at return to functioning and work with systems of care delivery. Clinical mental health research thus requires a multidisciplinary approach and a translational perspective. A research agenda exploring gaps and challenges, as well as the actions needed to address these challenges, is warranted. Therefore, the Clinical Research Task Force was established within ROAMER in order to perform such a survey, addressing these issues among experts in the field of clinical mental health research. The findings of the first round of this survey are presented in this paper.

Materials and methods

Procedure

The survey aimed to gain an insight into the challenges, gaps, and advances needed in clinical research in the field of mental disorders. It was performed among experts in this field and consisted of two rounds. The first round, performed from February to April 2013, aimed at setting priorities for mental health research, and the second round, performed from July to September 2013, focused on consensus among the experts. It was a web-based survey using Webropol 2.0 online survey and analysis software.21 Experts were asked to follow an individualized survey link to read the informed consent statement and to participate. The link was unique to each putative participant, and could not be forwarded to anyone else. Nonrespondents received reminder emails every third day and were approached by ambassadors and personal contacts from the ROAMER Clinical Research Taskforce in each country. The results of the first round were discussed during a scientific workshop in Amsterdam, March 18, 2013, in which the Clinical Research Task Force consulted experts in the field of clinical research on mental disorders. The current paper reports the results of the first round of the expert study as well as the results of the scientific workshop in Amsterdam. Analysis of the first round of the survey resulted in selection of main challenges as well as in prioritizing of the advances required to address these challenges, which are presented in the Results section. Discussions during the scientific workshop provided us with a broader perspective to interpret the findings of the first round of the survey, described in the Discussion section of this paper.

Selection of participants

In total, 313 experts in clinical mental health research were invited to participate, beginning on February 21, 2013 through April 8, 2013. They were selected by the members of the ROAMER Clinical Research Task Force according to one of the following criteria:

* minimum h-index of 10 in Web of Science,22 or more than 50 publications in the field in Web of Knowledge * experts who had been consulted in the Grand Challenges in Global Mental Health survey,23 who were scrutinized in duplicate (RK, CMFC) in order to identify all European experts involved in clinical research

* principal investigators carrying out clinical research in Europe were selected (RK, SK) from several databases for clinical research, such as the http://www.trialregister.nl/trialreg/index.asp trial register and the Web of Knowledge; the search terms used were in concordance with those used for other ROAMER activities and are shown in the Supplementary material

* from the list of experts already consulted by ROAMER in lieu of other surveys, the experts who published on clinical research topics in the Web of Knowledge in the last 5 years were selected (CMFC)

* ROAMER Clinical Research Task Force members were asked to list the most important experts from their own network using the same criteria

* in order to involve experts from as many European countries as possible, and thus attain sufficient representation in the survey with experts from all over Europe, members of ROAMER from countries with low representation, taking into account criteria 1 to 5, were asked to suggest experts meeting these criteria (DL, KW)

* ROAMER work package leaders involved in clinical research as well as the coordinator of ROAMER were also invited to participate.

Survey questions

The first round of the survey consisted of three sections: first, there was an open-ended section with questions on challenges and advances needed, followed by a section listing a priori challenges that the experts were asked to rate, and finally, an open section in which experts could propose advances that were required in order to address the challenges.

Section I: open question on challenges and advances

The expert survey started with a definition of a challenge in clinical research: "In line with previous initiatives, a challenge in clinical research is defined as "a specific barrier that, if removed, would help to solve an important health problem. If successfully implemented, the intervention(s) could lead to a high likelihood of feasibility for scaling up and impact."11 Subsequently, the experts were asked an open-ended question on perceived challenges in the field of clinical mental health research, as follows: "Please list your core challenge in Clinical Research on Mental Health and Well-Being in Europe – irrespective of any specific disease area or age group." The experts could list up to five such challenges.

Section II: a priori gaps and policies

In addition to the open-ended section on challenges, the first-round questionnaire also included 23 a priori general gaps, nine a priori methodological gaps, and ten a priori possible policies required to close those gaps identified by the Clinical Research Task Force and the ROAMER board, based on preliminary expert consultation (SK, CMFC). Experts participating in the survey were asked to rate these gaps using close-ended questions as follows. For the general gaps, the response could be chosen from: 1 (I do not see this as a gap), 2 (no priority), 3 (modest priority), and 4 (high priority). For the methodological gaps, the response could be either 1 (I do not see this as a gap) or 2 (I do see this as a gap). For the ten possible policies required to close gaps and advance challenges in clinical research, the possible responses were: 1 (I do not agree), 2 (I half-half), and 3 (I fully agree). Experts could not skip through or go back through the survey, so contamination of section 1 with items of section 2 could not occur and the rate of missing data was limited.

Section III: open questions on advances and their prioritization identified by experts

The last section contained open-ended questions prompting the experts to choose the three highest rated challenges from the first two sections, and to suggest advances to address these challenges. Input was provided by a member of the Scientific Advisory Board on how to conduct surveys and how to interpret their results.24

Analysis

The first open-ended question section resulted in a range of items (challenges) that survey participants evaluated as suitable for prioritization in clinical research. Items were categorized into twelve categories (themes) in duplicate and independently by two post-doctoral researchers (IE, SK). Duplicates and redundancies were removed (IE, SK). The Webropol survey program calculated means in the event of more than two-item scales, or percentages in the event of two-item scales, and provided the expert ratings on the respective a priori challenges and policy advances. Qualitative analysis of the data from the three sections of the questionnaire was done to identify the highest priority general challenges and advances for clinical research, and the highest priority methodological challenges, rated as such by the experts. These were subsequently discussed with the experts during the invitational conference.

Results

Participants

A total of 313 experts were invited for participation. Of these, 105 (34%) were identified as having accessed the system; an unknown number of experts did not receive the emails containing an invitation for and access to the survey. Of the 105 respondents identified to have accessed the website for the survey, 89 (response rate 85%; 28.4% of all those invited; 79% male respondents; mostly aged >50 years) from a variety of European countries participated in the first round of the survey. The respondents represented almost all European countries, with an overrepresentation of the Netherlands and the UK, which are the most productive countries in terms of clinical mental health research, particularly in randomized controlled trials (Figure 1).

Figure 1 European countries represented by the sample of respondents.

Respondents were mostly psychiatrists, psychologists, general physicians, and some occupational physicians who represented different areas in the field of clinical research. The best represented areas were clinical trials (71%), epidemiology and public health research (41%), health services research (34%), and basic research (30%). Experts represented different professional grades, disciplines, and institutions, with an overrepresentation of professors (74%) working at universities or in university hospitals (86%).

Challenges

Challenges identified regarding the open-ended questions in section I were reduced to 12 categories, and qualitative analysis showed substantial correspondence in challenges suggested by the experts in section I and the a priori-defined gaps and challenges in section II. Ratings of the experts for the predefined challenges therefore could be used to prioritize the combined challenges in sections I and II. The priority rating of the general challenges ranged from 2.6 to 3.6 in the 23 general items, and the total mean was 3.04. Items in the upper half above the mean rating of the priority ratings thus were items with a priority of 3.3 or more. These are shown in Table 1.

Table 1 General challenges for clinical mental health research Notes: Number of respondents: 87. Complete rating table experts first round: 23 items. Range 2.6–3.6.

The priority rating of the methodological challenges ranged from 2% to 73% in the eleven items. The mean percentage was 37%. The five items in the upper half of the priority ratings thus were items with a priority of over 37%. These are shown in Table 2.

Table 2 Methodological challenges for clinical mental health research Notes: Number of respondents: 84. Complete methodological rating table experts first round: 11 items. Range 2%–73%.

Advances

According to the experts, the challenge with the highest priority was: "to increase research on new intervention approaches in order to gain more insight into their working mechanisms and to successfully develop effective new interventions."

According to the experts, advances required to meet this challenge were: research on strategies in order to foster adherence to treatments or interventions; exploration of mediators and moderators of outcome; assessment of differential treatment effects, ie, if specific approaches are more effective for specific subgroups; development of e-Health/m-Health approaches and assessment of the level of human contact needed to motivate individuals toward sustained use of e-Health/m-Health treatments; and incorporation of patient perspectives in treatment.

The challenge rated as second in terms of priority was: "to conduct research in order to reach a wider understanding of different mechanisms (eg, psychological mechanisms, biological mechanisms, brain mechanisms, molecular mechanisms and environmental interactions) that may underlie diseases."

The challenge rated third in priority was: "to conduct research on different approaches to evaluate treatment effects." According to the experts, advances required to meet this challenge were of a more methodological nature, including standardization of psychotherapeutic treatment studies, equivalence trials, research on side effects of treatments, alternative and/or nonrandomized designs, and improved reliability and validity of outcome measures.

The challenge rated fourth in priority was: "to perform proof of concept clinical trials for innovative treatments." Experts indicated that advances of an infrastructural and methodological nature were required to meet this challenge and establish (European) research networks to coordinate and facilitate clinical research, and to identify or develop standard definitions and guidelines to increase the understanding of the term "proof of concept".

The experts were asked to indicate if specific patient groups, age groups, or disorders should be targeted, but in general there was no inclination toward a specific group, except the one defined in the fifth challenge. Experts identified the fifth challenge as the need for specific attention focused on comorbidity between somatic and mental disorders. The advances suggested by the experts to address this challenge were to develop and increase research in order to better understand the mechanisms of comorbidity between mental disorders and somatic conditions, including diagnostic strategies and interventions that target such comorbidity.

Policies

Finally, the three most prioritized policies indicated by the experts are shown in Table 3. These were an overall need for funding of randomized clinical trials in the field of clinical mental health research, the importance of Horizon 2020 in this respect, and adaption of outcome measures for research according to patients' expectations.

Table 3 Top three policies needed to meet the gaps in clinical research in Europe Notes: Number of respondents: 87. Complete rating table experts first round: 10 items. Range 2.2–2.77.

Discussion

Study limitations

The main limitation of this study is that it was based on a survey that was sent to 313 experts, of whom only 89 collaborated. Because of limitations imposed by spam filters in operation at universities, we can only be sure that 105 experts actually received the invitation for the survey, given that they accessed the system in order to indicate if they wished to collaborate, despite all efforts indicated earlier to contact experts in various ways. Thus, the response rate was 28%, which is within the range that can be expected for an email survey.

Study strengths

There were also some strengths in this survey. First, generalizability of the results may be good because the participating experts were representative of the target sample of all experts in Europe. Of all respondents, 79% were male, most were aged older than 50 years, and most were professors from a variety of European universities and university hospitals. Nearly all EU-28 countries were represented. Second, the number of experts may be considered high compared with most other studies describing expert opinions on research. Third, we surveyed a priori challenges using open-ended questions, providing an opportunity to suggest new areas, while avoiding contamination with the a prior section. Finally, it is also noteworthy that all data extraction was performed independently in duplicate by two researchers, and the findings were discussed with experts during the scientific workshop.

Main findings: need for new interventions

Three of the five priority items highlighted an overwhelming need for development and scientific evaluation of new interventions for mental disorders in the field of clinical mental health research. This type of research may be referred to as "experimental medicine", regardless of whether the mechanism under investigation is biological or psychological. The interventions regarded to have potential range from biologically oriented interventions such as pharmacological interventions and deep brain stimulation to psychotherapeutic, systemic, and social interventions. In addition, experts see potential for various kinds of treatments that use digital devices and/or information computer technology support, such as decision aids, m-Health, e-Health, and serious gaming. e-Health and m-Health interventions are mentioned by the experts, with the specification that research is required to improve sustained use of these interventions by patients.12 An important issue is the risk of health care provider withdrawal and patient disengagement found in e-Health research25–30 and the need to examine e-Health and m-Health as tools to augment the process of face-to-face care, rather than replace it.31 In Europe, industry is involved in information computer technology developments in the field of e-Health as well as m-Health, and this may provide Europe with opportunities to collaborate in this field in a precompetitive environment, combining efforts of companies and research in the context of Horizon 2020. In the same vein, the creation of a precompetitive combined research and industry environment may be needed in the context of Horizon 2020, given that many pharmaceutical companies have withdrawn from central nervous system research. Furthermore, new developments in neuroscience may open up opportunities for innovation in the future. This raises the question of to what degree governmental and independent funding will be able to develop new medications, possibly in collaboration with the pharmaceutical industry on an independent basis. A shift may be required in order to identify and test novel molecules in non-industry settings. Another challenge in the area of pharmacological interventions concerns the development of effective and safe algorithms for prescription of medication. For example, combinations of antipsychotic medications have rarely been tested, given that this type of research is rarely facilitated by the pharmaceutical industry, and is not routinely funded by the government either, whereas such research performed by independent clinical researchers may yield clinically relevant findings. For the same reason, head-to-head trials are also scarce, leaving many open questions on the relative benefit–risk of medicines for mental disorders.32

Another focus for research in this area concerns the study of possibly synergistic effects of combined psychopharmacological and psychotherapeutic interventions, given that the impact of pharmacological compounds is thought to be contingent on experience-based alterations in the brain and mental functioning in many ways. A more synergistic approach may also help to inform about the moderators, mediators, or even mechanisms of behavior and behavioral change.10

Furthermore, the emphasis on European growth in Horizon 2020 may support the indication by the experts of the importance of research into interventions aimed at return to work in sick-listed employees with a mental disorder; governmental organizations, trade unions, medical insurance companies, and social insurance companies may be relevant stakeholders in this effort.

Methodological advances needed for evaluation of treatment effects

A distinct need for innovation was identified, and in connection with this need, experts indicated that methodological advances are essential for intervention research. A need to develop methodology and design factors for proof-of-concept trials evaluating novel treatments was identified, eg, relatively small trials exploring the possible effect of a new intervention by assessment of intermediary outcomes.

Methodological issues are considered a priority in relation to the need for new psychotherapeutic interventions as well. In psychotherapeutic intervention studies, blinding problems are an issue. Studies are required to explore possibilities to work with placebo conditions in psychotherapeutic trials, and to explore the neurobiological underpinnings of placebo effects in psychotherapeutic interventions.33 Also, equivalence studies are needed as an alternative. Standardization of psychotherapy in research is an important issue, particularly in terms of replicability and internal integrity, as well as fidelity to treatment protocols. This problem, if properly addressed, could be translated to clinical practice by pragmatic trials exploring treatment adherence of patients and therapists alike. Also, in randomized clinical trials and meta-analytic reviews in clinical mental health research, particular attention should be paid to diminishing potential bias and resultant loss of external validity.34 It should also be taken into account that methodological advances are needed to translate findings derived from often well prepared and well conducted clinical research into clinical practice, ie, by mixed-methods designs combining pragmatic randomized clinical trials with qualitative research evaluating conditions facilitating implementation of effective interventions.

Patient-oriented clinical intervention research

Experts in clinical research indicated a need for patient-centered approaches. Among the advances suggested, the importance of tailoring treatments to specific patient profiles or diagnostic subgroups was mentioned as a high priority,35 as well as the importance of patient-oriented clinical mental health research based on outcomes that are clinically relevant to patients. Outcomes of clinical mental health research can be specified at the level of the main symptom, global treatment response, remission, or occurrence of relapse, but also at the level of quality of life and functioning, eg, in terms of work absenteeism in patients who are on sick leave. Also important as an outcome is societal participation other than work and the concept of personal recovery, reflecting the higher-order process of adaptation and self-management. Clinical mental health research should be sensitive to patient perspectives, and take into account patient preference in the design, experimental therapy, execution, and outcome specification of randomized clinical trials, and address adherence of doctors and therapists as well as that of patients with treatment protocols as process or outcome measures. Research may also take the patient perspective into account with use of shared decision-making methods. Both pilot studies36 and randomized clinical trials using such treatment models and patient preference designs37 have been performed in Europe with promising results.38 Clinical research also encompasses the introduction of methodology to assess patient stratification in order to account for and target specific subgroups based on etiology and/or treatment sensitivity, such as in the application of risk profiles. Both efficacy and pragmatic randomized clinical trials are required in order to establish actual effectiveness in clinical practice, as well as cost-effectiveness.

Concept and mechanisms of disease and somatic-psychiatric comorbidity

Among the five priorities identified by the experts, the first was the development and evaluation of new interventions, as described above. The other main emphasis, embedded in two of the five main challenges identified by the experts, was on what may be pertaining to current concepts of disease in psychiatry, particularly in terms of diagnosis and underlying mechanisms, as well as underlying mechanisms relevant for treatment, and the relationship between somatic and mental disorders.13 These were seen as areas for priority in clinical mental health research. There is wide consensus that diagnosis and classification in psychiatry, reflecting our current concepts of disease, is an area in need of improvement. Research in this area should result in the introduction of a system of functional diagnosis in psychiatry, ie, a system of diagnosis that is rich in information on the main functions that therapeutic interventions attempt to remediate.38 Insight into underlying mechanisms of etiology and particularly treatment are required to develop a system of functional diagnosis.24

Somatic-psychiatric comorbidity is frequent and associated with increased mortality.13 Management of these combined conditions is intrinsically difficult due to diagnostic, treatment-related,39 and organizational difficulties,40 and there is an urgent need for the development and scientific evaluation of new treatment modalities in this field, using both mental and somatic outcomes such as mortality and taking into account this diversity. From the perspective of the clinical research experts, research should address diagnostic and treatment complexities in somatic-psychiatric comorbidity, with a focus on novel diagnostic and treatment algorithms designed to support medical decision-making. This need is closely linked to the other advances needed in terms of our understanding of the mechanisms of disease in clinical mental health research. This brings us to a main problem indicated by the experts.

Subjectivity gap

Throughout the survey, experts indicated in different ways that there is little connection between much of the research in neuroscience, basic research, and imaging studies in psychiatry, and applied research in clinical practice, ie, both in clinical trials and in patients' experience of the signs and symptoms of their mental disorders, and how these are approached, interpreted, and treated by professionals. Experts identified this as a major problem in mental health research. This problem may be referred to as the "subjectivity gap: the genotyping, imaging or other preclinical studies do not provide input regarding subjective experience. Similarly, they do not relate to diagnostic criteria, and the clinical practice of psychiatry".

It has been noted that preclinical research to date has not yielded reliable diagnostic markers or markers for patient stratification in clinical practice.41 There is a need for clinical research addressing more precise and more patient-centered diagnostic assessments and patient-tailored interventions, which may lead to so-called stratified medicine.35 There is also a potential for novel, reliable, and specific biomarkers as well as novel clinical markers that inform diagnosis of mental disorder. There is a need for intensive time-sampling process-related clinical assessments with potential to model symptom level or behavioral level change in a dynamic system of mental experience. Clinical process assessments may be combined with brain-related parameters for the purposes of diagnosis and prediction of treatment response. A combined focus on biological markers and the mental experience of patients may bridge the subjectivity gap and enrich research in mental health.

Some of the methodological advances recommended by the experts may be related to this underlying conceptual problem. The assumption that neurobiological processes represent causes of mental disorders cannot be validated by research;8 the only relationship that may exist may be a parallel process of changes in social, mental, and neurobiological processes that can be induced by social, psychotherapeutic, and pharmacological interventions.8 Under that assumption, research is required that enables exploration of these perspectives in a synergistic and nonhierarchical fashion. This would enable researchers to operationalize the working mechanism of placebo as well as active conditions in psychotherapeutic interventions.16,42

If a link could be made between brain-oriented research and clinical mental health research, there should be no hierarchy between neurobiological, pharmacological, psychotherapeutic, and social paradigms in patient-related clinical research. Such an effort would require multidisciplinary teams that connect basic researchers, clinical researchers, and conceptual philosophers, as well as patients with mental disorders, providing the researchers with insights in specific needs as part of a translational effort. Ultimately, this may yield novel concepts of mental disorder. If this challenge is taken seriously by Europe, funding and support of clinical research networks working together with basic scientists in a translational manner is required. Given the predictions regarding the health burden of mental disorders,22 implementation of such networks at the European level would be productive.

The European clinical research priorities presented in this paper add to the Grand Challenges in Global Mental Health Initiative, which identified global mental health research priorities in the next ten years.23 The initiative lists top global mental health challenges, ranked by disease-burden reduction, impact on equity, immediacy of impact, and feasibility. In parallel with the ROAMER findings, the Global Mental Health Initiative rank list is also topped by research topics related to improving treatments, eg, on integration of mental health service provision in primary health care and improving children's access to evidence-based mental health care. Although methodological differences between the European and the global priority setting procedures preclude any firm comparative conclusions, it seems that European experts are more preoccupied with developing research methodology, such as classification, standardization, and promotion of proof-of-concept studies. Also, anecdotal evidence suggests that randomized clinical trials in the field of behavioral research are trailing behind in Europe in comparison with other countries, such as the USA. This type of clinical research, and the specific methodological attention required for innovation, eg, proof-of-concept trials, evaluation trials, pragmatic trials, and patient preference trials, tends to be overlooked and underfunded. Clinical mental health research in mental disorders should be brought more into line with other more medically oriented research. The difference between European and global priorities indicates that specific research needs exist in the European context and underlines the need for a specific European research agenda for mental health. Nevertheless, the fact that European Union mental health experts are putting together a Mental Health Horizon 2020 agenda for enhancing the well-being and happiness of people in the European Union in particular and the world at large, by providing evidence-based, better quality interventions for people suffering from mental disorders, may directly or indirectly provide other nations, including low-income, middle-income, and high-income countries, throughout the world with the opportunity to benefit from this planning and development framework. In the following second stage of the expert survey, we expect to focus further on how the experts expect to achieve the priorities and to analyze on which approaches consensus may be found.

The European Union research funding programs have been criticized for being too focused on biomedical research in the area of health, thus disregarding important translational research, such as implementation research, health systems and health policy research, and public health research.43,44 Indeed, in 2011, only 4% of the €642 million cooperation program for health research in 2011 was allocated to these fields.45 Our results indicate that randomized clinical trials in the field of mental disorders and translational and implementation research need a more visible position in the Horizon 2020 program than in previous European Union health research activities.

Conclusion

Innovations in clinical mental health research should bridge the gap between mechanisms underlying novel therapeutic interventions and patient experience of mental disorder and, if present, somatic comorbidity. Development of new interventions is needed, as well as studies evaluating these interventions, and pragmatic trials evaluating how to disseminate and translate basic research findings for application in clinical practice. The methodological advances needed for this purpose, as mentioned by the experts, should be developed whilst taking into account the need for enhanced conceptualization and research operationalization. The mechanisms underlying etiology and treatment response in mental disorders should be the subject of research as well as the relationship between somatic and mental comorbidity, with the purpose of translating this knowledge into clinical intervention research for mental disorders. Clinical mental health research is relatively underfunded and should receive specific attention in Horizon 2020 funding programs.

Acknowledgments

This study was performed as part of the ROAMER project: A road-map for mental health research in Europe. Coordination and Support Action, Objective Theme (HEALTH.2011.3.3-4). This project has received funding from the European Union Seventh Framework Program (FP7/2007-2013) under grant agreement number 282586, and from the National R&D Internationalisation Programme of the Spanish Ministry of Science and Technology under reference ACI-PRO-2011-1080. Our thanks are extended to: Jacques Demote-Mainard for providing input and supporting the work of the Clinical Research Task Force; Harold Pincus for providing suggestions on how to conduct the survey; Mario Maj, Istvan Bitter, and Patrick Boyer, for acting as ambassadors for the survey in their countries; Rebecca Kuepper for her work on selection of experts for the survey; Rosanna Wendel for her work on setting up the survey questionnaire; and Anna Forsman for her advice regarding the Webropol analysis. We also thank the 89 experts who participated in the survey, including the following who agreed to be identified: Cecilio álamo, Peter Allebeck, Alfredo Carlo Altamura, Celso Arango, Zerrin Atakan, Jose Luis Ayuso-Mateos, Ccorrado Barbui, Aartjan TF Beekman, Istvan Bitter, Filippo Bogetto, Peter Bower, Wim van den Brink, Wiepke Cahn, Somnath Chatterji, David Coghill, Francesco Colom, Pim Cuijpers, David McDaid, Daniel David, Jacques Demotes, Frank van Dijk, Guy Goodwin, Heinz Grunze, Beate Herpertz-Dahlmann, Lieuwe de Haan, Andreas Heinz, Marc de Hert, WJG Hoogendijk, Luigi Janirir, Ad JFM Kerkhof, Dimitrios Kontis, Ralph Kupka, Albert FG Leentjens, Francisco López-Muñoz, Mario Maj, Laura Mandelli, Donatella Marazziti, Massimo Carlo Mauri, Harm van Marwijk, CL Mulder, David Nutt, Ketil J Oedegaard, MJ Owen, Kirsi Peltonen, Andreas Reif, Aribert Rothenberger, Aart HSchene, Robert ASchoevers, Alessandro Serretti, Michael Sharpe, Wolfgang Soellner, Dollfus Sonia, Philip Spinhoven, Jan Spijker, David Taylor, Robert JM Vermeiren, Jan Walburg, Nick JA van der Wee, Jaap van Weeghel, and Mathias Zink. We are also grateful to the following experts who participated in the invitational conference on the results of round one of the expert survey: Istvan Bitter, Matthias Brunn, Jan Buitelaar, David McDaid, Louise Gallagher, Mario Luciano, Marta Miret, Szilvia Papp, and Siegfried Weyerer.

Author contributions

This research is published on behalf of the ROAMER consortium. The Clinical Research Task Force was led by CMFC. CMFC, JvO, and SK designed the survey. JvO provided input for the first part of the survey. SK and CMFC devised the predefined challenges in the second part of the survey, and DL and JvO provided input for this part as well. SK and IE analyzed the survey data in duplicate. CMFC presented the data at the Scientific Workshop. CMFC wrote the paper, and JvO, EV, SK, GS, H-UW, JMH, SWL, CO-T, DL, IE and KW contributed and approved the final version.

Disclosure

CMFC has received unrestricted grants for investigator-initiated research and unrestricted grants for lectures from Eli Lilly. JvO has received unrestricted investigator-led research grants or recompense for presenting his research from Eli Lilly, Bristol-Myers Squibb, Lundbeck, Organon, Janssen-Cilag, GlaxoSmithKline, AstraZeneca, Pfizer, and Servier, companies that have an interest in the treatment of psychosis. SWL has been a consultant or made educational presentations for Janssen-Cilag, Abbvie and AstraZeneca. EV has received grants and served as consultant, advisor, or speaker for the following entities: Alexza, Almirall, AstraZeneca, Bristol-Myers Squibb, Cephalon, Elan, Eli Lilly, Ferrer, Forest Research Institute, Gedeon Richter, GlaxoSmithKline, Janssen-Cilag, Jazz, Johnson and Johnson, Lundbeck, Merck, Novartis, Organon, Otsuka, Pfizer, Roche, Sanofi-Aventis, Servier, Schering-Plough, Shire, the Spanish Ministry of Science and Innovation, the Seventh European Framework Programme, the Stanley Medical Research Institute, Sunovion, Takeda, Teva, United BioSource Corporation, and Wyeth. JMH has been a consultant or made educational presentations for Eli Lilly and Company and Lundbeck, and has served on advisory boards for Eli Lilly and Company, Lundbeck, AstraZeneca, and Hoffman-La Roche, Inc. The other authors report no conflicts of interest in this work.

Supplementary material

Search terms for experts Delphi survey

Web of Science

Search History – "ROAMER Delphi survey"

#10 #6 AND #5 Refined by: Document Types=(REVIEW) AND Web of Science Categories=(PSYCHIATRY OR PHARMACOLOGY PHARMACY OR PSYCHOLOGY CLINICAL OR PSYCHOLOGY OR FAMILY STUDIES OR PUBLIC ENVIRONMENTAL OCCUPATIONAL HEALTH OR PSYCHOLOGY DEVELOPMENTAL OR HEALTH POLICY SERVICES OR PSYCHOLOGY MULTIDISCIPLINARY OR PSYCHOLOGY BIOLOGICAL OR REHABILITATION OR PSYCHOLOGY APPLIED OR HEALTH CARE SCIENCES SERVICES OR PSYCHOLOGY PSYCHOANALYSIS) AND [excluding] Publication Years=(2013)

DocType=All document types; Language=All languages;

#9 #6 AND #5 Refined by: Document Types=(REVIEW) AND Web of Science Categories=(PSYCHIATRY OR PHARMACOLOGY PHARMACY OR PSYCHOLOGY CLINICAL OR PSYCHOLOGY OR FAMILY STUDIES OR PUBLIC ENVIRONMENTAL OCCUPATIONAL HEALTH OR PSYCHOLOGY DEVELOPMENTAL OR HEALTH POLICY SERVICES OR PSYCHOLOGY MULTIDISCIPLINARY OR PSYCHOLOGY BIOLOGICAL OR REHABILITATION OR PSYCHOLOGY APPLIED OR HEALTH CARE SCIENCES SERVICES OR PSYCHOLOGY PSYCHOANALYSIS)

DocType=All document types; Language=All languages;

#8 #6 AND #5 Refined by: Document Types=(REVIEW)

DocType=All document types; Language=All languages;

#7 #6 AND #5 DocType=All document types; Language=All languages;

#6 #4 AND #3 DocType=All document types; Language=All languages;

#5 #2 OR #1 DocType=All document types; Language=All languages;

#4 TS=(clinical* OR trial OR psychiatry OR psychology OR psychiatr* OR psychol*)

DocType=All document types; Language=All languages;

#3 TS=(("drug abuse" OR "drug addict" OR "drug addicts" OR "drug addiction" OR "drug addicted" OR "drug dependent*" OR "drug dependence*" OR "drug withdrawal" OR "drug abuse") OR ("addictive disease*" OR "addictive disorder*") OR ("alcoholic patient*" OR "alcoholic subject*" OR alcoholism OR "alcohol dependent*" OR "alcohol dependence*" OR "fetal alcohol*" OR "prenatal alcohol*" OR "chronic ethanol*" OR "chronic* alcohol*" OR "alcohol withdrawal" OR "ethanol withdrawal") OR ("caffeine dependent*" OR "caffeine dependence" OR "caffeine addiction" OR (caffeine AND addict*) OR "caffeine withdrawal") OR (((cocaine OR heroin OR cannabis OR MDMA OR ecstasy OR morphine*) AND (abuse OR depend* OR dependent* OR dependence* OR addict* OR addicts OR addicted OR addiction* OR withdrawal)) OR methadone) OR (addiction OR addictive OR "substance abuse" OR "withdrawal syndrome" OR psychoactive*) OR ((schizophrenia OR schizophrenic) OR Schizotyp* OR ((Delusional OR paranoid) AND disorder*) OR hallucination* OR Psychotic OR Schizoaffective OR psychosis) OR (((manic OR bipolar OR mood) AND disorder*) OR (depressive AND (disorder* OR episode*)) OR "depressive symptom*" OR hypomania OR mania* OR ((major OR psychotic OR disorder*) AND depression) OR "suicide attempt*" OR suicidal* OR cyclothymia OR Dysthymia) OR (((anxiety OR panic OR "Obsessive-compulsive" OR adjustment OR conversion OR dissociative OR Somatoform OR Somatization OR neurotic) AND disorder*) OR ("hypochondriasis*" OR "body dysmorphic disorder*" OR "pain disorder*") OR agoraphobia OR "social phobia*" OR "Post-traumatic stress" OR "stress disorder*") OR ("Eating disorder*" OR "Anorexia nervosa" OR "Bulimia nervosa" OR "sleep disturbance" OR (sexual AND (disorder* OR dysfunction)) OR ((postnatal OR postpartum) AND depression) OR ((antidepressant* OR laxative* OR analgesic* OR psychotropic* OR vitamin* OR steroids OR hormone*) AND abuse)) OR (((insomnia OR sleepiness OR "sleep disturbance") NOT (apnea OR "side effect*" OR parkinson* OR alzheimer OR neurodegenerat* OR cancer OR obesity OR obese*)) OR (hypersomnia NOT narcolepsy) OR ((sleep OR night) AND terror*) OR nightmare*) OR ((disorder* AND (personality OR identity OR impulse* OR impulsive* OR impulsivity)) OR asocial OR antisocial OR psychopathic OR anxious OR narcissi* OR "Pathological gambling" OR pyromania* OR Trichotillomania OR Psychosexual OR ("Munchhausen syndrome")) OR ("Pervasive developmental disorder*" OR autism OR autistic* OR "Rett* syndrome" OR "Asperger* syndrome") OR (((Hyperkinetic OR Conduct OR Emotional OR tic) AND disorder*) OR (anxiety AND (separation OR phobic OR social)) OR (hyperactivity AND (disorder* OR syndrome)) OR "Tourette syndrome" OR "Tourette's syndrome") OR ((Mental AND (disorder* OR illness OR health)) OR "psychological distress" OR "psychiatric disorder") OR (Nervousness OR "nervous tension" OR Irritability) OR anorexia OR (neurosis OR neuroses OR psychoses) OR (("mental confusion*") OR ("mental disability*") OR ("mental capacity*") OR ((psychiatric OR mental) AND (comorbidity OR comorbid)) OR psychiatry OR psychology))

DocType=All document types; Language=All languages;

#2 GP=(Europe* OR "EU-27" OR Nordic* OR Austria* OR Belgium OR Belgian OR Flemish OR Bulgaria* OR Cyprus OR Cypriot* OR Czech* OR Denmark OR Danish OR Estonia* OR Finland OR Finish OR France OR French OR German* OR Greece OR Greek OR Hungary OR Hungarian OR Ireland OR Irish OR Italy OR Italian OR Latvia* OR Lithuania* OR Luxembourg OR Malta OR Maltese OR Netherland* OR Dutch OR Holland* OR Poland OR Polish OR Portugal OR Portuguese OR Romania* OR Slovak* OR Slovania* OR Spain OR Spanish OR Sweden OR Swedish OR "United Kingdom" OR "Great Britain" OR British OR Britannic OR England OR English OR Wales OR Welsh OR Scotland OR Scottish OR Croatia* OR Macedonia* OR Iceland* OR Montenegr* OR Turkey OR Turkish OR Albani* OR Andorra OR Armeni* OR Azerbaijan* OR Azerbaidzhan* OR Belarus* OR Byelarus* OR Bosnia* OR Georgia* OR Liechtenstein OR Moldova OR Moldovan OR Monaco OR Monacan OR Monegasque OR Norway OR Norwegian OR Russia* OR "San Marino" OR Serbia* OR Switzerland OR Swiss OR Ukrain* OR Vatican*)

DocType=All document types; Language=All languages;

#1 CU=(Europe* OR "EU-27" OR Nordic* OR Austria* OR Belgium OR Belgian OR Flemish OR Bulgaria* OR Cyprus OR Cypriot* OR Czech* OR Denmark OR Danish OR Estonia* OR Finland OR Finish OR France OR French OR German* OR Greece OR Greek OR Hungary OR Hungarian OR Ireland OR Irish OR Italy OR Italian OR Latvia* OR Lithuania* OR Luxembourg OR Malta OR Maltese OR Netherland* OR Dutch OR Holland* OR Poland OR Polish OR Portugal OR Portuguese OR Romania* OR Slovak* OR Slovania* OR Spain OR Spanish OR Sweden OR Swedish OR "United Kingdom" OR "Great Britain" OR British OR Britannic OR England OR English OR Wales OR Welsh OR Scotland OR Scottish OR Croatia* OR Macedonia* OR Iceland* OR Montenegr* OR Turkey OR Turkish OR Albani* OR Andorra OR Armeni* OR Azerbaijan* OR Azerbaidzhan* OR Belarus* OR Byelarus* OR Bosnia* OR Georgia* OR Liechtenstein OR Moldova OR Moldovan OR Monaco OR Monacan OR Monegasque OR Norway OR Norwegian OR Russia* OR "San Marino" OR Serbia* OR Switzerland OR Swiss OR Ukrain* OR Vatican*)

DocType=All document types; Language=All languages;

1. World Health Organization. Global status report on non-communicable diseases. Geneva, Switzerland: World Health Organization; 2010. Available from: http://www.who.int/nmh/publications/ncd_report_full_en.pdf. Accessed March 27, 2014.

2. Wittchen HU, Jacobi F, Rehm J, et al. The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011;21:655–679.

3. MHEEN Group. Employment and mental health: Assessing the economic impact and the case for intervention. 2008. Available from: http://eprints.lse.ac.uk/4236/1/MHEEN_policy_briefs_5_Employment(LSERO).pdf. Accessed March 24, 2014.

4. Murray CJ, Vos T, Lozano R, et al. Disability-adjusted life years (DALYs) for 291 diseases and injuries in 21 regions,1990–2010: a systematic analysis for the Global Burden of Disease Study. Lancet. 2012;380:2197–2223.

5. Gustavsson A, Svensson M, Jacobi F, et al; CDBE 2010 Study Group. Cost of disorders of the brain in Europe 2010. Eur Neuropsychopharmacol. 2011;21:718–779.

6. Vos T, Flaxman AD, Naghavi M, et al. Years lived with disability (YLDs) for 1160 sequelae of 289 diseases and injuries 1990–2010: a systematic analysis for the Global Burden of Disease Study 2010. Lancet. 2012;380:2163–2196.

7. Haro JM, Ayso-Mateos JL, Bitter I, et al. ROAMER: a European roadmap for mental health research. Int J Methods Psychiatr Res. 2014; 23 Suppl 1:1–13.

8. Forsman AK, Ventus DBJ, van der Feltz-Cornelis CM, et al. Public mental health research in Europe: a systematic mapping for the ROAMER Project. Eur J Public Health. 2014 in press.

9. Wittchen H-U, Knappe S, Schumann G. The psychological perspective on mental health and mental disorder research. Int J Methods Psychiatr Res. 2014;23 Suppl 1:15–27.

10. Wittchen H-U, Knappe S, Andersson G, et al. The need for a behavioural science focus in research on mental health and mental disorders. Int J Methods Psychiatr Res. 2014;23 Suppl 1:28–40.

11. Goschke T. Dysfunctions of decision-making and cognitive control as transdiagnostic mechanisms of mental disorders: advances, gaps, and needs in current research. Int J Methods Psychiatr Res. 2014; 23 Suppl 1:51–57.

12. Emmelkamp PMG, David D, Beckers T, et al. Advancing psychotherapy and evidence-based psychological interventions. Int J Methods Psychiatr Res. 2014;23 Suppl 1:58–91.

13. Fava GA, Tossani E, Bech P, et al. Emerging clinical trends and perspectives of comorbid patterns of mental disorders. Int J Methods Psychiatr Res. 2014;23 Suppl 1:92–101.

14. Fiorillo A, Luciano M, Del Vecchio V, Sampogna G, Obradors-Tarragó C, Maj M; on behalf of the ROAMER Consortium. Priorities for mental health research in Europe: a survey among national stakeholders’ associations within the ROAMER project. World Psychiatry. 2013;12: 165–170.

15. Priebe S, Bruns T, Craig TKJ. The future of academic psychiatry may be social. Br J Psychiatry. 2013;202:319–320.

16. van der Feltz-Cornelis CM, Sarchiapone M, Postuvan V, et al. Best practice elements of multilevel suicide prevention strategies: a review of systematic reviews. Crisis. 2011;32:319–333.

17. While D, Bickley H, Roscoe A, et al. Implementation of mental health service recommendations in England and Wales and suicide rates,1997–2006: a cross-sectional and before-and-after observational study. Lancet. 2012;379:1005–1012.

18. Van der Feltz-Cornelis CM. Ten years of integrated care for mental disorders in The Netherlands. Int J Integr Care. 2011;11 Spec Ed:e015.

19. Van der Feltz-Cornelis CM, Van Os TW, Van Marwijk HW, et al. Effect of psychiatric consultation models in primary care. A systematic review and meta-analysis of randomized clinical trials. J Psychosom Res. 2010;68:521–533.

20. Archer J, Bower P, Gilbody S, et al. Collaborative care for depression and anxiety problems. Cochrane Database Syst Rev. 2012;10:CD006525.

21. Webropol. 2.0 online survey and analysis software. Available from: http://www.webropol.com/. Accessed March 27, 2014.

22. Hirsch JE. An index to quantify an individual’s scientific research output. Proc Natl Acad Sci U S A. 2005;102:16569–16572.

23. Collins PY, Patel V, Joestl SS, et al; Scientific Advisory Board and Executive Committee of the Grand Challenges on Global Mental Health. Grand challenges in global mental health. Nature. 2011;475:27–30.

24. Scholle SH, Pincus HA. Survey research: think... think again. Acad Psychiatry. 2003;27(2):114–116.

25. Black AD, Car J, Pagliari C, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011;8:e1000387.

26. Balas EA, Krishna S, Kretschmer RA, et al. Computerized knowledge management in diabetes care. Med Care. 2004;42:610–621.

27. Takahashi PY, Pecina JL, Upatising B, et al. A randomized controlled trial of telemonitoring in older adults with multiple health issues to prevent hospitalizations and emergency department visits. Arch Intern Med. 2012;172:773–779.

28. Steventon A, Bardsley M, Billings J, et al; Whole System Demonstrator Evaluation Team. Effect of telehealth on use of secondary care and mortality: findings from the Whole System Demonstrator cluster randomised trial. BMJ. 2012;344:e3874.

29. Car J, Huckvale K, Hermens H. Telehealth for long term conditions. BMJ. 2012;344:e4201.

30. Chavannes NH, Sont JK, Van der Boog JKM, et al. [E-health for the chronically ill. Not in every situation and not fit for everybody]. Ned Tijdschr Geneeskd. 2012;156:A5345. Dutch.

31. Kramer I, Simons C, Hartmann JA, et al. A therapeutic application of momentary assessment in the treatment of depression: a randomized controlled trial. World Psychiatry. 2014;13(1):68–77.

32. Vieta E, Cruz N. Head to head comparisons as an alternative to placebo-controlled trials. Eur Neuropsychopharmacol. 2012;22:800–803.

33. Benedetti F, Amanzio M. Mechanisms of the placebo response. Pulm Pharmacol Ther. 2013;26:520–523.

34. Ioannidis JP. Why most published research findings are false. PLoS Med. 2005;2:e124.

35. Schumann G, Binder EB, Holte A, et al. Stratified medicine for mental disorders. Eur Neuropsychopharmacol. 2014;24:55–60.

36. Van der Feltz-Cornelis CM, Andrea H, Kessels E, et al. [What can be done with routine outcome monitoring? A clinical-empirical exploration of shared decision making in combination with ROM in patients with somatic symptom disorder]. Tijdschr Psychiatr. 2014 in press. Dutch.

37. Le QA, Doctor JN, Zoellner LA, et al. Minimal clinically important differences for the EQ-5D and QWB-SA in post-traumatic stress disorder (PTSD): results from a Doubly Randomized Preference Trial (DRPT). Health Qual Life Outcomes. 2013;11:59.

38. Huijbregts KM, de Jong FJ, van Marwijk HW, et al. A target-driven collaborative care model for Major Depressive Disorder is effective in primary care in the Netherlands. A randomized clinical trial from the depression initiative. Journal of Affective Disorders. 2013;146(3):328–337.

39. Fleischhacker WW, Cetkovich-Bakmas M, De Hert M, et al. Comorbid somatic illnesses in patients with severe mental disorders: clinical, policy, and research challenges. J Clin Psychiatry. 2008;69:514–519.

40. Van der Feltz-Cornelis CM, Ten Have M, Penninx BW, et al. Presence of comorbid somatic disorders among patients referred to mental health care in The Netherlands. Psychiatr Serv. 2010;61:1119–1125.

41. Naylor C, Parsonage M, McDaid D, et al. Long-term conditions and mental health: the cost of co-morbidities. London, UK: The King’s Fund; 2012. Available from: http://www.kingsfund.org.uk/publications/long-term-conditions-and-mental-health. Accessed March 27, 2014.

42. Kapur S, Phillips AG, Insel TR. Why has it taken so long for biological psychiatry to develop clinical tests and what to do about it? Mol Psychiatry. 2012;17:1174–1179.

43. Wood AM, Joseph S. An agenda for the next decade of psychotherapy research and practice. Letter to the Editor. Psychol Med. 2010;40: 1055–1056.

44. McCarthy M. Public health research – Europe’s future (STEPS report). London, UK: University College London; 2011. Available from: http://discovery.ucl.ac.uk/1329165. Accessed March 27, 2014.

45. Walshe K, McKee M, McCarthy M, et al. Health systems and policy research in Europe: Horizon 2020. Lancet. 2013;382:668–669.

Christina M van der Feltz-Cornelis,1–3 Jim van Os,4,6 Susanne Knappe,5 Gunter Schumann,6 Eduard Vieta,7 Hans-Ulrich Wittchen,5 Shôn W Lewis,8 Iman Elfeddali,2,9 Kristian Wahlbeck,10,11 Donald Linszen,4 Carla Obradors-Tarragó,12,13 Josep Maria Haro12–14

1Trimbos Instituut, Utrecht, 2Tilburg University, Tranzo Department, Tilburg, 3GGz Breburg, Tilburg, 4Department of Psychiatry and Psychology, South Limburg Mental Health Research and Teaching Network, Euron, Maastricht University Medical Center, Maastricht, the Netherlands; 5Institute of Clinical Psychology and Psychotherapy and Center for Epidemiology and Longitudinal Studies, Technische Universität Dresden, Dresden, Germany; 6Institute of Psychiatry, King’s College London, London, UK; 7Institute of Neuroscience, Hospital Clinic, University of Barcelona, IDIBAPS, CIBERSAM, Barcelona, Catalonia, Spain; 8School of Community-Based Medicine, University of Manchester, Manchester, UK; 9Department of Health Promotion/School of Public Health and Primary Care, Maastricht University, Maastricht, the Netherlands; 10The Nordic School of Public Health, Gothenburg, Sweden; 11National Institute for Health and Welfare, Helsinki, Finland; 12Centro de Investigación Biomédica en Red de Salud Mental, Madrid, 13Research and Development Unit, Parc Sanitari Sant Joan de Déu, Fundació Sant Joan de Déu, Sant Boi de Llobregat, 14Universitat de Barcelona, Barcelona, Spain

Word count: 7997

Show less

© 2014. This work is licensed under https://creativecommons.org/licenses/by-nc/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.

Abstract

Background: The size and increasing burden of disease due to mental disorders in Europe poses substantial challenges to its population and to the health policy of the European Union. This warrants a specific research agenda concerning clinical mental health research as one of the cornerstones of sustainable mental health research and health policy in Europe. The aim of this research was to identify the top priorities needed to address the main challenges in clinical research for mental disorders.

Methods: The research was conducted as an expert survey and expert panel discussion during a scientific workshop.

Results: Eighty-nine experts in clinical research and representing most European countries participated in this survey. Identified top priorities were the need for new intervention studies, understanding the diagnostic and therapeutic implications of mechanisms of disease, and research in the field of somatic-psychiatric comorbidity. The “subjectivity gap” between basic neuroscience research and clinical reality for patients with mental disorders is considered the main challenge in psychiatric research, suggesting that a shift in research paradigms is required.

Conclusion: Innovations in clinical mental health research should bridge the gap between mechanisms underlying novel therapeutic interventions and the patient experience of mental disorder and, if present, somatic comorbidity. Clinical mental health research is relatively underfunded and should receive specific attention in Horizon 2020 funding programs.

Details

Title
Towards Horizon 2020: challenges and advances for clinical mental health research – outcome of an expert survey
Author
Christina M van der Feltz-Cornelis; Jim van Os; Knappe, Susanne; Schumann, Gunter; Vieta, Eduard; Wittchen, Hans-Ulrich; Lewis, Shôn W; Elfeddali, Iman; Wahlbeck, Kristian; Linszen, Donald; Obradors-Tarragó, Carla; Haro, Josep Maria
Pages
1057-1068
Section
Original Research
Publication year
2014
Publication date
2014
Publisher
Taylor & Francis Ltd.
ISSN
1176-6328
e-ISSN
1178-2021
Source type
Scholarly Journal
Language of publication
English
DOI
https://doi.org/10.2147/NDT.S59958
ProQuest document ID
2240083940
Copyright
© 2014. This work is licensed under https://creativecommons.org/licenses/by-nc/3.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.