Towards the development of a comprehensive framework: Qualitative systematic survey of definitions of clinical research quality

Abstract

Objective

To systematically survey existing definitions, concepts, and criteria of clinical research quality, both developed by stakeholder groups as well as in the medical literature. This study serves as a first step in the development of a comprehensive framework for the quality of clinical research.

Study design and setting

We systematically and in duplicate searched definitions, concepts and criteria of clinical research quality on websites of stakeholders in clinical research until no further insights emerged and in MEDLINE up to February 2015. Stakeholders included governmental bodies, regulatory agencies, the pharmaceutical industry, academic and commercial contract research organizations, initiatives, research ethics committees, patient organizations and funding agencies from 13 countries. Data synthesis involved descriptive and qualitative analyses following the Framework Method on definitions, concepts, and criteria of clinical research quality. Descriptive codes were applied and grouped into clusters to identify common and stakeholder-specific quality themes.

Results

Stakeholder concepts on how to assure quality throughout study conduct or articles on quality assessment tools were common, generally with no a priori definition of the term quality itself. We identified a total of 20 explicit definitions of clinical research quality including varying quality dimensions and focusing on different stages in the clinical research process. Encountered quality dimensions include ethical conduct, patient safety/rights/priorities, internal validity, precision of results, generalizability or external validity, scientific and societal relevance, transparency and accessibility of information, research infrastructure and sustainability. None of the definitions appeared to be comprehensive either in terms of quality dimensions, research stages, or stakeholder perspectives.

Conclusion

Clinical research quality is often discussed but rarely defined. A framework defining clinical research quality across stakeholders’ individual perspectives is desirable to facilitate discussion, assessment, and improvement of quality at all stages of clinical research.

About the Authors:

Belinda von Niederhäusern

* E-mail: [email protected]

Affiliation: Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Stefan Schandelmaier

Affiliations Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, Basel, Switzerland, Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

Marie Mi Bonde

Affiliation: Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Nicole Brunner

Affiliation: Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Lars G. Hemkens

Affiliation: Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Marielle Rutquist

Affiliation: Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Neera Bhatnagar

Affiliation: Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

Gordon H. Guyatt

Affiliation: Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

Christiane Pauli-Magnus

¶‡ CPM and MB are joint senior authors on this work.

Affiliation: Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, Basel, Switzerland

Matthias Briel

¶‡ CPM and MB are joint senior authors on this work.

Affiliations Basel Institute for Clinical Epidemiology and Biostatistics, Department of Clinical Research, University Hospital Basel, Basel, Switzerland, Department of Health Research Methods, Evidence, and Impact, McMaster University, Hamilton, Ontario, Canada

Introduction

Clinical research is necessary to advance our knowledge and practice of diagnosing and preventing diseases and treating patients. However, its complexity and the regulatory requirements have significantly increased over the last few years, requiring an ever-rising level of scientific, methodological, regulatory and organizational know-how [1]. Global clinical research involves billions of dollars and millions of people, yet it is often poorly planned, inefficient, or “not useful”, leading to considerable waste of private and public funding [1–8]. Low quality research may not only result in misleading findings [9], but may also compromise safety and rights of patients.

The regulatory international “ethical and scientific quality standard for designing, conducting, recording and reporting trials”–the Good Clinical Practice (GCP) guideline developed by the International Conference on Harmonisation (ICH) aims to ensure that safety and rights of participants are protected and that trial data are credible [10, 11]. The GCP guideline is a widely disseminated and applied standard for the broad concept of clinical research quality. However, its limitations include development as an agreement between industry and regulatory experts and its focus on data accuracy and extensive formal requirements has been criticized as an unsuitable standard for investigator-initiated clinical research [12, 13]. The GCP guidelines lack a broad stakeholder consensus and a sound evidence-base [14, 15].

In academic clinical research, “quality” often relates to design and implementation from the standpoint of scientific rigor. Over the last two decades a large number of quality assessment instruments and checklists have focused on specific aspects of quality in the context of specific types of research (e.g. the Cochrane Risk of Bias (RoB) tool for randomized trials [16], the tool for Quality Assessment of Diagnostic Studies (QUADAS-2) [17], or the Risk Of Bias In Non-randomized Studies tool (ROBINS-I) [18]). Other instruments have addressed the reporting of results from specific study types (e.g. CONsolidated Standards for Reporting Trials (CONSORT) [19], STrengthening the Reporting of OBservational studies in Epidemiology (STROBE) [20], or Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [21]) and accordingly the reporting of protocols (e.g. Standard Protocol Items: Recommendations for Interventional Trials (SPIRIT) [22]). The Grading, Recommendation, Assessment, Development, and Evaluation (GRADE) initiative addresses risk of bias and, in addition, imprecision, inconsistency, and indirectness as domains to assess the overall quality of a “body of evidence” for the development of evidence-based clinical guidelines [23]. These instruments and checklists are useful means to address specific aspects of quality but do not consider the research process itself.

In other research fields, including higher education [24], legal sciences [25], or political sciences [26], the assessment of overall research quality has been described as complex, ambiguous, and a “major issue”. Increasingly, efforts have been directed towards the development of comprehensive quality frameworks [27]. Such broader approaches to quality assessment should consider the extent to which research meets the needs and expectations of stakeholders, and therefore depends on their perspective. However, the stakeholders in clinical research are numerous and their particular interests and priorities differ. Measurements of quality of clinical research may therefore be limited, or distorted, if prior consensus on a definition of quality has not been reached, and if the complexity of clinical research itself and the variety of stakeholders involved has not been taken into account. Avedis Donabedian, a pioneer in the assessment of the quality of care, declared in 1980: “What is missing (…) is a unifying theory of the definition and measurement of quality of care” (…) Before we attempt to assess the quality of care, either in general terms or in any particular site or situation, it is necessary to come to an agreement on what the elements that constitute it are” [28, 29].

This study aims to provide an overview of the existing definitions, concepts, and criteria of clinical research quality and to examine their variability by systematically synthesizing qualitative sources from the involved stakeholder groups and the medical literature. Clinical research in this context is defined as research conducted with patients to answer therapeutic, preventive, diagnostic, or prognostic questions or investigations of the mechanisms of human disease. We explicitly exclude research focusing on health care system processes, structures or policies (such as health services research or health technology assessments) and research with healthy volunteers. The findings of this study will inform the next step, i.e. the composition and structure of a comprehensive framework for clinical research quality as a common goal to increase value and reduce waste.

Methods

We conducted two systematic searches for definitions, concepts, and criteria of clinical research quality (see Box 1 for definitions of terms). We searched (i) websites and any linked documents of stakeholders in clinical research, and (ii) the published medical literature.

Box 1. Glossary of working definitions, in alphabetical order

Clinical Research

Interventional and observational research addressing health care issues and involving human participants.

Concept of quality

An implicit statement on what clinical research quality means and comprises, e.g. which criteria are needed to ensure good quality research (often operational, e.g. “at our institution, the factors required to ensure quality are…“), or a discussion of one or multiple quality dimensions in the context of clinical research (e.g. internal validity, external validity, transparency, etc.).

Definition of quality

An explicit statement on what clinical research quality means and comprises, e.g. “quality of clinical research may be defined as the internal validity of study results and their applicability to patient treatment”, “quality of clinical research is commonly defined as…”, or “we define quality as…”. May include one or multiple quality dimensions.

Quality criteria

Aspects that are described as integral part(s) of quality, e.g. adherence to guidelines, use of standard operating procedures, etc.

Quality dimension

Overarching categories of quality criteria, e.g. internal validity, external validity, relevance, transparency, etc.

Quality framework

Theoretical foundation for a definition or concept of quality spanning multiple dimensions and study phases in a matrix structure; and serving the development of quality indicators for operationalization.

Quality indicator

An instrument to assess or measure an individual quality criterion, a group of quality criteria, or a quality dimension, i.e. the operationalization of quality criteria or dimensions (e.g. how to assess the adherence to guidelines).

Quality theme

A recurrent topic in the qualitative analysis of text material about quality definitions, concepts, or criteria extracted from stakeholder websites or articles published in the literature.

Search of stakeholder websites

Stakeholder website selection.

We searched stakeholder organizations (national ministry of health, regulatory body, pharmaceutical industry association, academic research organization, ethics committee, patients’ organization, funding agency, and initiative for clinical research) in 13 countries (Australia, Austria, Canada, France, Germany, Italy, Japan, Norway, Spain, Sweden, Switzerland, USA, UK) to provide perspectives from developed nations in different geographic regions. To identify at least one representative national stakeholder organization per stakeholder category in each of the 12 countries, we used personal contacts to one recognized expert in clinical research or public health per country. For the two contacts that did not respond (Australia, Norway), we identified the national organizations for all categories through a web search. We additionally searched for websites of inter- or supranational bodies involved in clinical research (e.g. ICH, WHO, Horizon2020, international associations) and the global 2013 Top10 pharmaceutical companies (IMSExecutive) and Contract Research Organizations (pharma-iq.com). We eventually identified publicly available websites of 155 organizations using the Google Search Engine (see S1 Table for the full list of screened organizations).

Eligibility criteria and search process.

We systematically and in duplicate screened each website for a statement on a definition or concept of quality by the respective organization (e.g. “our trials are of high quality because they matter to patients”, or “quality means relevant, valid, and ethical trials”) using the keywords “quality” or “good” and “clinical research” or “clinical studies” or “clinical trials” or “research” in the website’s search function. If we did not find a statement on quality, we extended the search to related website content, e.g. “our policy”, “what we do”, “standards & quality assurance” etc., as well as organizational statements, guidelines, and reports. Within these documents we repeated the search for the above search terms using the respective search function. If no statements were found through the search function, the text was manually searched for paragraphs that described either a) the standards according to which the organization performed clinical research (i.e. ICH-GCP, Declaration of Helsinki, etc.), b) criteria according to which the organization assesses the quality of clinical research (e.g. evaluation criteria of funding programs), c) the processes used to assure the quality of clinical research within an organization (e.g. “quality assurance procedures”), or d) criteria which a “good study” should fulfil within the organization. We did not consider any statements that focused on animal research, quality of life, or quality of health care without providing any definition related to clinical research. For websites presented in languages other than English or German, text passages were translated by members of the investigative team (BvN, CPM, MMB, MR).

Search of the literature (MEDLINE)

With the help of an experienced research librarian (NB) we designed a comprehensive search strategy using MeSH terms and text words (see S1 Text for full search strategy) and conducted a systematic literature search in MEDLINE using the Ovid interface from database inception to February 27, 2015. We did not impose any language restrictions.

Eligibility criteria and selection process.

We included any article describing a definition, a concept, criteria, or a checklist, guideline, or measurement instrument of quality spanning more than one quality dimension of clinical research in general or within a specific clinical discipline. We excluded any articles not suggesting a definition, concept, or criteria of clinical research quality (e.g. exclusively discussing the implementation or validation of individual quality criteria or guidelines without providing any definition related to clinical research), systematic reviews applying an assessment tool of a specific aspect of quality (e.g. systematic reviews on the reporting quality of trials in a specific field applying CONSORT [19], or articles suggesting a measurement instrument/assessment tool of one specific aspect of quality (e.g. the Jadad Scale [30]). In addition, we excluded articles that focused on animal research, quality of life, or quality of health care without providing any definition related to clinical research.

Working in pairs, methodologically trained reviewers applied the pre-defined eligibility criteria independently after undergoing a calibration process. The reviewers first screened titles and abstracts. If titles and abstracts suggested an article meeting the above mentioned inclusion criteria or if eligibility remained unclear, we obtained corresponding full texts. Disagreements were resolved by discussion and consensus.

Data extraction

We designed standardized extraction sheets suitable for qualitative data extraction (S2 Table) accompanied by an instruction manual. Before starting data extraction, the data extraction forms were piloted and teams of reviewers conducted calibration exercises to ensure consistency. We extracted text sections on the definition, concept, or criteria of quality from both literature and internet sources independently and in duplicate. Data synthesis of included articles involved categorization by overall topic, author, year of publication, article citation index (as retrieved in ISI Web of Science by 11 January 2016), and journal name. Internet sources were categorized by stakeholder group, country, and name of organization.

Data analysis

We performed descriptive and qualitative explanatory analyses following the Framework Method [31] on definitions, concepts, and criteria of clinical research quality stratified by stakeholders and on evaluation criteria of funding agencies for clinical studies. The Framework Method belongs to a family of qualitative approaches termed thematic or content analysis, which identify commonalities and differences in qualitative data, and eventually seek to draw descriptive and/or explanatory conclusions clustered around themes. Its defining feature is the matrix output, i.e. rows (cases), columns (codes) providing a structure into which the researcher can systematically reduce the data in order to analyze it [32]. We therefore applied codes to excerpts of raw data and added or modified as new responses emerged. Codes were then grouped into clusters around similar and interrelated ideas to identify common and stakeholder-specific quality themes in an iterative process until consensus between the three investigators (BvN, MB, CPM) was reached. Themes were named after the most frequently recurring terms within the same clusters (e.g. generalizability, relevance, high quality data etc.) and were not created or imposed by the investigators.

Results

Definitions or concepts of clinical research quality in different stakeholder groups

We screened publicly available websites and linked documents of 155 stakeholders. Concepts of how to assure quality of clinical research or quality criteria were commonly reported among most stakeholder groups (66.4% (103/155); i.e. in 86.1% (31/36) of pharmaceutical companies or contract research organizations (CROs), 72% (18/25) of academic research organizations or initiatives, 63.6% (14/22) of international and governmental organizations, 61.9% (13/21) of regulatory agencies, 57.9% (11/19) of ethics committees, and 63.2% (12/19) of funding agencies, respectively), but this was relatively uncommon for patient organizations (31%; 4/13). However, only 12 of 155 (7.7%) institutions provided an explicit definition of the term ‘clinical research quality’ (pharmaceutical companies or CROs: 3/36; academic research organizations and initiatives: 3/25; international and governmental organizations: 3/22; regulatory agencies: 2/21; patient organizations: 1/13; ethics committees and funding agencies: 0/38) (S3 Table).

Qualitative analysis of the 12 definitions and the 103 quality concepts or criteria resulted in both common and stakeholder-specific quality themes often focusing on different stages of clinical research (planning/feasibility, conduct, dissemination; Table 1). Common quality themes amongst stakeholder groups included the adherence to all applicable national and international laws and regulations (e.g. ICH GCP), a scientific and methodologically rigorous approach allowing for an efficient and effective answer to the research question, credible and high quality data, the inclusion of trained study personnel, and the presence of Standard Operating Procedures (SOPs) and monitoring.

[Figure omitted. See PDF.]

Table 1. Qualitative analysis of common and stakeholder-specific quality themes in the context of clinical research.

https://doi.org/10.1371/journal.pone.0180635.t001

Stakeholder-specific emphasis on quality themes ranged from “high quality data” (pharmaceutical industry and CROs); “adherence to guidelines” (regulatory agencies); “patient involvement and applicability of research” (patient organizations); “absence of bias, relevance, and transparency” (academic research and/or initiatives); to “feasibility, generalizability, and objectivity of research” (funding agencies). The terminology used by the stakeholders to describe these themes (e.g. relevance, transparency, feasibility), was no less open to definition than the overarching concept of “quality” and as well depends on the perspective of the observer. In general, priorities within stakeholder groups were similar across different countries. However, for national funding agencies we found considerable variation in quality criteria that were particularly emphasized as relevant for funding decisions across countries (S4 Table).

Definitions or concepts of clinical research quality in the medical literature

Our systematic MEDLINE search yielded 8’289 titles and abstracts, of which we reviewed 90 articles in full text (Fig 1). We excluded 43 full text articles from detailed analysis, because they did not discuss a definition, concept, or criteria of quality (n = 18), they were systematic overviews/summaries of existing quality assessment checklists, instruments, or scores, with or without critical discussions of their validity and/or reliability (n = 5), or they discussed specific measurement instruments of a single dimension of quality (n = 20).

[Figure omitted. See PDF.]

Fig 1. Article flow diagram.

https://doi.org/10.1371/journal.pone.0180635.g001

We included the remaining 47 articles for more detailed analysis (S2 Text). These provided concepts on how to assure or improve overall clinical research quality in specific contexts (e.g. at an academic institution, in a specific country, in a specific industry setting, or in a specific medical field; n = 18), or how to improve quality assessment (e.g. of RCTs, in radiology or hepatology research; n = 6). Measurement instruments or checklists that spanned more than one quality dimension were reported in 23 articles. A large proportion of these tools provided indicators on how to assess bias (n = 21). Almost half of them covered indicators on precision (n = 16), external validity (n = 16), or reporting quality (n = 14). Some tools additionally covered innovation aspects (n = 8) or ethical considerations (n = 4). None of the reviewed articles provided a definition or concept of clinical research quality spanning the encountered range of quality dimensions reflected by stakeholder perspectives such as ethical conduct, patient safety, patient values and preferences, absence of bias, precision, external validity, relevance, generalizability, transparency, infrastructure, and sustainability. Furthermore, we could not identify a definition or concept simultaneously covering several dimensions and differentiating between consecutive stages of research (e.g. study planning, conduct and dissemination), independent of a specific medical field or study setting.

Overall, we identified eight (8.9%) of 90 articles that provided an explicit definition of the term ‘clinical research quality’ (Table 2). The definitions therein span quality from methodological dimensions such as internal validity, external validity, or precision, and operational criteria including adherence to guidelines and applicable regulations (ethical conduct), to the effect of research at the societal level (relevance). None of the definitions appeared to be comprehensive either in terms of quality dimensions, research stages, or stakeholder perspectives. Five of the eight articles were cited less than 10 times in ISI Web of Science^™ by 11.01.2016 (Table 2).

[Figure omitted. See PDF.]

Table 2. Characteristics of articles providing an explicit definition of clinical research quality; by author (n = 8).

https://doi.org/10.1371/journal.pone.0180635.t002

Discussion

Summary of findings

Our systematic review of stakeholder websites and the medical literature showed that quality of clinical research is frequently discussed, but rarely defined. Although stakeholder groups seem to agree on a basic concept of quality, their emphasis in the conceptualization of clinical research quality varies widely. The medical literature contains many articles discussing approaches to measurement or assessments of quality without prior definition of the term itself, and without reflecting the diversity of stakeholder needs, interests and expectations. A major proportion of these identified quality assessments aim to evaluate the “methodological rigor of randomized controlled trials”. The definition of “methodological rigor” in itself, however, varies substantially between the reported tools. Most authors suggested assessing methodological quality based on the presence or absence of measures to prevent bias. Others included dimensions such as external validity, reporting, or relevance of the study in question. We did not, however, identify a definition or concept including multiple dimensions or differentiating between consecutive stages of research across medical fields, or study settings. Although a comprehensive “definition” of quality may be difficult, a “concept” or “framework” of research quality, rather than a “definition”, could span all research stages and include more than an assessment focused on one aspect of quality. A more comprehensive approach to quality assessment, i.e. ranging from conceptualization to dissemination of a study as proposed by the authors of the Lancet series on “increasing value, reducing waste” [1, 4–7], rather than evaluation of the final published product, would assist in identification of errors that matter at earlier stages, and therefore support reduction of research “waste” more efficiently.

Strengths and limitations

To our knowledge this is the first systematic survey addressing definitions and concepts of clinical research quality. Our systematic approach was suited to detect knowledge gaps, and to examine overlap and differences in perspectives of clinical research quality across stakeholder groups. Further strengths of this study include our consideration of websites and any linked documents from a large number of stakeholders in 13 different countries in addition to a Medline search. Methodologically trained investigators screened articles and websites in duplicate following a pre-specified instruction manual and undergoing a calibration process.

We acknowledge the following limitations: Although we consider our approach comprehensive, we searched only Medline as electronic database and relied on search terms in the title, abstract or other records. Articles in journals not indexed in Medline or providing some definition of research quality in the main text only might have been overlooked. However, Medline covers the most impactful journals and articles in current medical research and articles specifically focusing on quality of clinical research most likely mention this prominently. We may have missed definitions on websites despite screening these in duplicate. We would, however, expect stakeholder groups in clinical research to be transparent and proactive in defining such an important cornerstone of their activities, similarly to efforts in the field of clinical care quality. When coding our findings from the website search as well as from the Medline search we felt that we reached saturation, i.e. the last excerpts from websites or journal articles on aspects of clinical research quality did not bring new insights. The coding and qualitative analysis naturally involved subjective judgments of investigators, which we controlled by performing analyses in triplicate (BvN, CPM, MB) and comparing codes, findings, and interpretations until we reached consensus. Further, we acknowledge that our survey solely portrays perceptions on the quality of research in high income nations that may not necessarily overlap with those of low- or middle-income countries. With a growing percentage of clinical research being conducted in these geographies, a further study investigating quality perceptions of local stakeholders taking into consideration societal aspects and beliefs would be of importance. Finally, we did not conduct a detailed survey of experts or stakeholder groups—except for national funding agencies—nor did we conduct interviews with representatives of these groups to explore reasons for the paucity of explicit definitions.

Comparison with other studies and implications

We are not aware of any other systematic survey on definitions, concepts, or criteria of overall clinical research quality. A similar approach has been taken by other authors to develop a framework for excellence, however with a distinct focus on translational cancer research [41]. In the field of health care quality, the focus of assessments has more and more shifted from process-based measurements towards the evaluation of patient outcomes and patient satisfaction [42–44]. Clinical research conducted in this context of “patient-centered” care would explicitly warrant the engagement and involvement of patients in setting priorities. However, patients (or their representatives) are only rarely considered when discussing the quality of research that might impact their care [45–47]. In our analysis, we also found patients to be surprisingly underrepresented. First, patient organizations were among the last in providing definitions or concepts of clinical research quality. Second, only six (14%) of a total of 43 quality measurement tools or assessments covered an item on patient safety and/or rights (Fig 1). Most efforts in quality assessments so far were taken to ensure compliance with guidelines, methodologically rigorous designs and valid study results. While this may ultimately serve the treatment of disease, we were expecting the clinical research enterprise to put unmet medical need and applicability to the patient population first. Compared with medical care, the clinical research machinery still seems to function with relatively low engagement of the end user (patients) of the product.

Furthermore, while we expected variations in the perception of quality across stakeholder groups, we were surprised how different and vaguely defined some of the concepts were. For example, an explicit definition of “high quality data” may be as dependent on the perspective of the observer as the definition of “high quality research”. It may be linked to concepts such as relevance of the data and absence of errors in the data or the way the data is collected. Similarly, the quality criteria used by funding agencies such as “impact”, “relevance”, or “feasibility” varied in their clarity and elaboration. Public funding agencies have a major role in terms of defining what and how research topics are investigated. Those who use these criteria to evaluate proposals are still left with subjective interpretation, while applicants may aim to provide the readers with these buzzwords with not much reflection on their meaning.

There remains considerable ambiguity in the use of current quality criteria across and within stakeholder groups. Unless carefully explained, these concepts can be easily misinterpreted by the stakeholders. Finding consensus on a common definition or concept of clinical research quality across national borders, stakeholder groups, and study types may therefore seem arduous; assessments of methodological quality do not, however, suffice. Existing quality guidelines such as ICH GCP have not been developed based on consensus across the full range of stakeholder groups, but only between regulatory experts and industry [15]. Existing quality assessment tools predominantly cover single aspects of quality, or particular research stages. Furthermore, there is a lack of approaches tailored to stakeholder requirements in assessing the quality of clinical research, e.g. from a patient’s perspective on how to choose a “good trial”, or from a funding agency’s perspective on how to assess the quality of studies before, during, and after the funding period. The authors of a follow-up study to the 2014 Lancet series reported that academic institutions in particular had paid only little attention to their recommendations on how to increase value in research. Practical guidance on how to implement these recommendations is so far lacking and urgently needed to increase value of academic research at all stages.

Conclusions

This systematic survey serves as a first step of evidence summary to inform the development of a comprehensive framework of clinical research quality. It showed that definitions of clinical research quality are rarely provided and the existing definitions fall short of a theoretical or empirical framework across different study designs and stages and considering the variety of stakeholders involved. Based on our findings, a practically applicable framework needs to include the encountered quality dimensions such as ethical conduct, patient safety/rights/priorities, internal validity, precision of results, generalizability or external validity, scientific and societal relevance, transparency and accessibility of information, research infrastructure and sustainability) and consider different study stages such as planning, conduct, and dissemination. We plan to circulate framework drafts amongst stakeholder representatives of all eight groups until consensus on structure and content is reached, and to operationalize the framework through the development of instruments guiding stakeholder groups (e.g. academic institutions or funding agencies) in the comprehensive quality assessment of the full clinical research continuum.

Supporting information

[Figure omitted. See PDF.]

S1 Table. List of all screened stakeholder organizations (n = 155).

https://doi.org/10.1371/journal.pone.0180635.s001

(DOCX)

S2 Table. Data extraction forms.

https://doi.org/10.1371/journal.pone.0180635.s002

(DOCX)

S3 Table. Quality definitions found through systematic internet search; by institution.

Total number of institutions screened = 155.

https://doi.org/10.1371/journal.pone.0180635.s003

(DOCX)

S4 Table. Quality themes appearing in proposal evaluation criteria of funding agencies.

https://doi.org/10.1371/journal.pone.0180635.s004

(DOCX)

S5 Table. PRISMA 2009 Checklist.

https://doi.org/10.1371/journal.pone.0180635.s005

(DOC)

S1 Text. Literature search strategy.

https://doi.org/10.1371/journal.pone.0180635.s006

(DOCX)

S2 Text. References of eligible articles.

https://doi.org/10.1371/journal.pone.0180635.s007

(DOCX)

Acknowledgments

We would like to thank Dr. Priya Satalkar for her guidance and support with the qualitative analysis.

Author Contributions

1. Conceptualization: BvN CPM MB GHG.

2. Data curation: N. Bhatnagar.

3. Formal analysis: BvN CPM MB.

4. Investigation: BvN CPM MB SS LGH MMB N. Brunner MR.

5. Methodology: BvN CPM MB GHG.

6. Writing – original draft: BvN CPM MB.

7. Writing – review & editing: BvN CPM MB SS LGH MMB MR GHG.

Citation: von Niederhäusern B, Schandelmaier S, Mi Bonde M, Brunner N, Hemkens LG, Rutquist M, et al. (2017) Towards the development of a comprehensive framework: Qualitative systematic survey of definitions of clinical research quality. PLoS ONE 12(7): e0180635. https://doi.org/10.1371/journal.pone.0180635

References

1. Al-Shahi Salman R, Beller E, Kagan J, Hemminki E, Phillips RS, Savulescu J, et al. Increasing value and reducing waste in biomedical research regulation and management. Lancet. 2014;383(9912):176–85. Epub 2014/01/15. pmid:24411646.

2. Kasenda B, von Elm E, You J, Blumle A, Tomonaga Y, Saccilotto R, et al. Prevalence, characteristics, and publication of discontinued randomized trials. Jama. 2014;311(10):1045–51. Epub 2014/03/13. pmid:24618966.

3. Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gulmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet. 2014;383(9912):156–65. Epub 2014/01/15. pmid:24411644.

4. Chan AW, Song F, Vickers A, Jefferson T, Dickersin K, Gotzsche PC, et al. Increasing value and reducing waste: addressing inaccessible research. Lancet. 2014;383(9913):257–66. Epub 2014/01/15. pmid:24411650.

5. Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet. 2014;383(9913):267–76. Epub 2014/01/15. pmid:24411647.

6. Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet. 2014;383(9912):166–75. Epub 2014/01/15. pmid:24411645.

7. Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JP, et al. Biomedical research: increasing value, reducing waste. Lancet. 2014;383(9912):101–4. Epub 2014/01/15. pmid:24411643.

8. Ioannidis JP. Why Most Clinical Research Is Not Useful. PLoS Med. 2016;13(6):e1002049. Epub 2016/06/22. pmid:27328301.

9. Juni P, Altman DG, Egger M. Systematic reviews in health care: Assessing the quality of controlled clinical trials. Bmj. 2001;323(7303):42–6. pmid:11440947.

10. The International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use, ICH Harmonised Tripartite Guideline: Guideline for Good Clinical Practice. https://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R1_Guideline.pdf. Accessed 11 Nov 2016.

11. International conference on harmonisation of technical requirements for registration of pharmaceuticals for human use, Intergrated Addendum to ICH E6(R1): Guideline for Good Clinical Practice E6R(2). http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Efficacy/E6/E6_R2__Step_4.pdf. Accessed 11 Nov 2016.

12. Stewart PM, Stears A, Tomlinson JW, Brown MJ. Regulation—the real threat to clinical research. Bmj. 2008;337:a1732. Epub 2008/10/18. pmid:18927171.

13. McMahon AD, Conway DI, Macdonald TM, McInnes GT. The unintended consequences of clinical trials regulations. PLoS Med. 2009;3(11):e1000131. Epub 2009/11/18. pmid:19918557.

14. Woolf SH. Practice guidelines, a new reality in medicine. II. Methods of developing guidelines. Arch Intern Med. 1992;152(5):946–52. Epub 1992/05/01. pmid:1580720.

15. Grimes DA, Hubacher D, Nanda K, Schulz KF, Moher D, Altman DG. The Good Clinical Practice guideline: a bronze standard for clinical research. Lancet. 2005;366(9480):172–4. pmid:16005342.

16. Higgins JP, Altman DG, Gotzsche PC, Juni P, Moher D, Oxman AD, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. Bmj. 2011;343:d5928. Epub 2011/10/20. pmid:22008217.

17. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med. 2011;155(8):529–36. Epub 2011/10/19. pmid:22007046.

18. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Bmj. 2016;355:i4919. Epub 2016/10/14. pmid:27733354.

19. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134(8):663–94. pmid:11304107.

20. Vandenbroucke JP, von Elm E, Altman DG, Gotzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the Reporting of Observational Studies in Epidemiology (STROBE): explanation and elaboration. Int J Surg. 2014;12(12):1500–24. Epub 2014/07/22. pmid:25046751.

21. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. Ann Intern Med. 2009;151(4):W65–94. Epub 2009/07/23. pmid:19622512.

22. Chan AW, Tetzlaff JM, Gotzsche PC, Altman DG, Mann H, Berlin JA, et al. SPIRIT 2013 explanation and elaboration: guidance for protocols of clinical trials. Bmj. 2013;346:e7586. http://dx.doi.org/10.1136/bmj.e7586. pmid:23303884.

23. Guyatt GH, Oxman AD, Kunz R, Vist GE, Falck-Ytter Y, Schunemann HJ. What is "quality of evidence" and why is it important to clinicians? Bmj. 2008;336(7651):995–8. Epub 2008/05/06. pmid:18456631.

24. Patrick WJ, Stanley EC. Assessment of Research Quality. Research in Higher Education. 1996;37(1):23–42.

25. Castermans AG. Evaluating Dutch Legal Research Quality Assessment. Evaluating Dutch Legal Research Quality Assessment. 2015:447–73. RS_6researchqualityassessment447researchqualityassessment.

26. Kellow A. Assessing Political Science Quality: ‘Excellence in Research for Australia’. European Political Science. 2012;11(4):567–80.

27. Cabral AP, Huet I. Assessment of Research Quality in Higher Education: Contribution for an Institutional Framework. Procedia—Social and Behavioral Sciences. 2014;116:1528–32. http://dx.doi.org/10.1016/j.sbspro.2014.01.429.

28. Donabedian A. The Definition of Quality and Approaches to Its Assessment. Ann Arbor: Health Administration Press. 1980;Volume I: Explorations in Quality Assessment and Monitoring (2):176 pp.;.

29. Donabedian A. The quality of care. How can it be assessed? Jama. 1988;260(12):1743–8. Epub 1988/09/23. pmid:3045356.

30. Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJ, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials. 1996;17(1):1–12. pmid:8721797.

31. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13(1):1–8. pmid:24047204

32. Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. 2013;13:117. Epub 2013/09/21. pmid:24047204.

33. Moher D, Jadad AR, Tugwell P. Assessing the quality of randomized controlled trials. Current issues and future directions. Int J Technol Assess Health Care. 1996;12(2):195–208. pmid:8707495.

34. Verhagen AP, de Vet HC, de Bie RA, Boers M, van den Brandt PA. The art of quality assessment of RCTs included in systematic reviews. J Clin Epidemiol. 2001;54(7):651–4. pmid:11438404.

35. Njie VP, Thomas AC. Quality issues in clinical research and the implications on health policy (QICRHP). J Prof Nurs. 2001;17(5):233–42. pmid:11559879.

36. Franck LS, Pendleton E, Pittam B, Preece M, Aynsley-Green A. Quality assurance for clinical research: challenges in implementing research governance in UK hospitals. Int J Health Care Qual Assur Inc Leadersh Health Serv. 2004;17(4–5):239–47. pmid:15481690.

37. Switula D. The concept of quality in clinical research. Sci Eng Ethics. 2006;12(1):147–56. pmid:16501656.

38. Krestin GP. Evaluating the Quality of Radiology Research: What Are the Rules of the Game? Radiology. 2008;249(2):418–24. pmid:18936307

39. Bhatt A. Quality of clinical trials: A moving target. Perspect Clin Res. 2011;2(4):124–8. pmid:22145122.

40. Balshem H, Helfand M, Schunemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401–6. http://dx.doi.org/10.1016/j.jclinepi.2010.07.015. pmid:21208779.

41. Rajan A, Caldas C, van Luenen H, Saghatchian M, van Harten WH. Assessing excellence in translational cancer research: a consensus based framework. J Transl Med. 2013;11:274. Epub 2013/10/31. pmid:24168073.

42. Jenkinson C, Coulter A, Bruster S. The Picker Patient Experience Questionnaire: development and validation using data from in-patient surveys in five countries. Int J Qual Health Care. 2002;14(5):353–8. Epub 2002/10/23. pmid:12389801.

43. Urden LD. Patient satisfaction measurement: current issues and implications. Lippincotts Case Manag. 2002;7(5):194–200. Epub 2002/10/24. pmid:12394558.

44. Dawn AG, Lee PP. Patient satisfaction instruments used at academic medical centers: results of a survey. Am J Med Qual. 2003;18(6):265–9. Epub 2004/01/14. pmid:14717384.

45. Dugas M, Trottier M, Chipenda Dansokho S, Vaisson G, Provencher T, Colquhoun H, et al. Involving members of vulnerable populations in the development of patient decision aids: a mixed methods sequential explanatory study. BMC Med Inform Decis Mak. 2017;17. pmid:28103862.

46. Coulter A, Stilwell D, Kryworuchko J, Mullen PD, Ng CJ, van der Weijden T. A systematic development process for patient decision aids. BMC Med Inform Decis Mak. 2013;13(Suppl 2):S2. pmid:24625093.

47. Frank L, Forsythe L, Ellis L, Schrandt S, Sheridan S, Gerson J, et al. Conceptual and practical foundations of patient engagement in research at the patient-centered outcomes research institute. Qual Life Res. 2015;24(5):1033–41. pmid:25560774.