Meta-Review of the Quantity and Quality of Evidence for Knee Arthroplasty Devices

Abstract

Introduction

Some cardiovascular devices are licensed based on limited evidence, potentially exposing patients to devices that are not safe or effective. Research is needed to ascertain if the same is true of other types of medical devices. Knee arthroplasty is a widely-used surgical procedure yet implant failures are not uncommon. The purpose of this study was to characterize available evidence on the safety and effectiveness of knee implants.

Methods

A review of primary studies included in health technology assessments (HTA) on total (TKA) and unicompartmental knee arthroplasty (UKA) was conducted. MEDLINE, EMBASE, CINAHL, Cochrane Library and Biotechnology & BioEngineering Abstracts were searched from 2005 to 2014, plus journal tables of contents and 32 HTA web sites. Patients were aged 18 and older who underwent primary TKA or UKA assessed in cohort or randomized controlled studies. Summary statistics were used to report study characteristics.

Results

A total of 265 eligible primary studies published between 1986 and 2014 involving 59,217 patients were identified in 10 HTAs (2 low, 7 moderate, 1 high risk of bias). Most evaluated TKA (198, 74.5%). The quality of evidence in primary studies was limited. Most studies were industry-funded (23.8%) or offered no declaration of funding or conflict of interest (44.9%); based on uncontrolled single cohorts (58.5%), enrolled fewer than 100 patients (66.4%), and followed patients for 2 years or less (UKA: single cohort 29.8%, comparative cohort 16.7%, randomized trial 25.0%; TKA: single cohort 25.0%, comparative cohort 31.4%, randomized trial 48.6%). Furthermore, most devices were evaluated in only one study (55.3% TKA implants, 61.1% UKA implants).

Conclusions

Patients, physicians, hospitals and payers rely on poor-quality evidence to support decisions about knee implants. Further research is needed to explore how decisions about the use of devices are currently made, and how the evidence base for device safety and effectiveness can be strengthened.

About the Authors:

Anna R. Gagliardi

* E-mail: [email protected]

Affiliation: University Health Network, Toronto, Canada

Ariel Ducey

Affiliation: University of Calgary, Calgary, Canada

Pascale Lehoux

Affiliation: University of Montreal, Montreal, Canada

Sue Ross

Affiliation: University of Alberta, Edmonton, Canada

Patricia Trbovich

Affiliation: University Health Network, Toronto, Canada

Anthony Easty

Affiliation: University of Toronto, Toronto, Canada

Chaim Bell

Affiliation: Mount Sinai Hospital, Toronto, Canada

Julie Takata

Affiliation: University Health Network, Toronto, Canada

Christof Pabinger

Affiliation: Medical University of Innsbruck, Innsbruck, Austria

David R. Urbach

Affiliation: University Health Network, Toronto, Canada

Introduction

Medical decision-making is meant to be informed by the best available evidence, clinical judgment, and patient values and preferences. However it appears that what constitutes evidence may differ between drug and non-drug technologies. While pharmaceutical products must undergo years of rigorous testing, analysis of evidence for high-risk cardiovascular devices approved by the United States Food and Drug Administration found that the quantity and quality of pre- and post-market studies was lacking, potentially exposing patients to devices that were not safe and effective [1,2]. The same was true of metal-on-metal total hip replacement, which was associated with high revision rates, and subsequent analysis of explanted components found that they had been modified from the manufacturer’s specifications [3].

Before advocating for broad changes in the policies and processes of pre- and/or post-market surveillance, further studies are needed to ascertain if the same is true of other types of medical devices. Knee arthroplasty is among the most common and effective procedure currently performed [4], and is expected to increase in frequency [5]. However, surgical complications and implant failures are not uncommon, with ten-year revision rates of 6.2% for total (TKA), and 16.5% for unicompartmental knee arthroplasty (UKA) [6,7].

No research has fully described the evidence that payers, hospitals, physicians and patients must rely on when making decisions about knee arthroplasty. It was hypothesized that, similar to studies of cardiovascular devices [1–3], evidence on the safety and effectiveness of knee implants may be limited due to issues of randomization, blinding, and the expense of measuring long-term outcomes [5]. The purpose of this study was to characterize the nature of the available evidence regarding the safety and effectiveness of knee arthroplasty devices. Specifically, this study sought to describe limitations of studies that evaluated knee implants.

Methods

Approach

This study described the limitations of studies that evaluated knee arthroplasty devices; it did not seek to assess if knee implants are clinically effective as that research has been done [4]. A meta-review was conducted of primary studies included in health technology assessments (HTAs) of knee arthroplasty devices. HTA is defined as the systematic evaluation of properties, effects and/or impacts of health technologies and interventions including intended and unintended consequences [6]. We used the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) criteria (S1 Checklist) [7]. A protocol for this review was not registered. Institutional review board approval was not necessary.

Eligibility

Inclusion and exclusion criteria were initially generated based on the Patients, Intervention, Comparison and Outcomes (PICO) framework, and used to search and screen for HTAs, and then to screen primary studies included in those reviews. Patients included adults aged 18 and older from any country who underwent knee arthroplasty for any indication. The intervention of interest was primary total (TKA) or unicompartmental knee arthroplasty (UKA). Comparisons included single cohort studies evaluating a device, either before and after, or only after surgery; or comparative cohort studies or randomized trials comparing patients before and after, or only after receiving different types of devices. Studies varied in the outcomes they reported. To include studies that evaluated the safety and effectiveness of devices while favouring inclusion, eligible studies reported at least two of the following most frequently reported outcomes: complications (surgical or device-specific), revision rate (absolute number or device survival), or functional outcomes (e.g. pain, health status, quality of life, ability to complete physical tasks, satisfaction) assessed either by clinicians or patients using standardized instruments. Searches were limited to English language. Publications in the form of editorials, protocols, abstracts, or proceedings were not eligible. Studies were not eligible if they focused on evaluating the effectiveness of a surgical approach (i.e. minimally invasive, computer-aided) or technique (e.g. posterior, lateral or anterior approach; type of incision, sutures, instrumentation, bone cement), or on rehabilitation interventions or quality of life following surgery. This review focused on HTAs because HTAs are a form of evidence that is readily available to the majority of health care professionals to inform real-time decisions about which devices to purchase and use; and include primary studies of pre- and post-market evaluation upon which regulatory licensing decisions are made. Studies based on registry data were excluded. Although such studies provide useful data due to the large number of included patients, they do so only after a considerable period of time during which devices were licensed and used in many patients.

Searching and screening

Knee arthroplasty HTAs were identified in MEDLINE, EMBASE, CINAHL, Health Technology Assessment database in the Cochrane Library, and Biotechnology & BioEngineering Abstracts. These were searched on January 14, 2015 from 2005 to 2014 inclusive. The search strategy (S1 Table) was purposefully broad to be as inclusive as possible. We also searched the tables of contents of Health Technology Assessment and the International Journal of Technology Assessment in Health Care, and 35 web sites of HTA agencies (S2 Table). Titles and abstracts were independently screened by three reviewers. All items selected by at least one reviewer were retrieved. Then the primary studies included in HTAs were screened. If two or more primary studies were based on the same cohort of patients, the single most recent or complete study was eligible and the outcomes it reported were included.

Data extraction

Data were extracted from primary studies on author, country, year published, HTA source, arthroplasty type (UKA, TKA) and device (model, company). To identify limitations of the studies that evaluated knee implants, data were extracted on study design, number of patients included in final analyses, years of follow-up and conflict of interest (independent, industry funded, undetermined). We did not extract outcome data; the effectiveness of knee arthroplasty has already been established (4). ARG and two trained research assistants independently pilot-tested data extraction on the same three articles and compared findings through two iterations at which time data extraction was congruent. Two research assistants extracted data from remaining studies. ARG independently checked data to resolve discrepancies or other issues.

Data analysis

The methodological quality of HTAs was assessed using the Assessing the Methodological Quality of Systematic Reviews (AMSTAR) instrument [8]. Each study was scored for the presence of 11 elements, and the total score was categorized as high (0 to 4), moderate (5 to 8) and low (9 to 11) risk of bias. Summary statistics were used to describe the number of studies by country, year of publication, and type of implant. The methodological quality of primary studies was described with summary statistics for study design, number of participants, length of follow-up and potential conflicts of interest.

Results

Search results

The PRISMA diagram appears in Fig 1. Ten HTAs were eligible [9–19]. Those HTAs included 346 primary studies, of which 35 were duplicates, 46 were excluded (29 no device evaluation, 9 non-English language, 8 registry studies), leaving 265 primary studies that were included in the review.

[Figure omitted. See PDF.]

Fig 1. PRISMA diagram.

https://doi.org/10.1371/journal.pone.0163032.g001

HTA characteristics

Of the 10 HTAs, 2 had a low risk of bias, 7 had a moderate risk of bias and 1 had a high risk of bias (Table 1, based on S3 Table). Two HTAs were issued in Canada and the United States, and one each in Austria, Australia, Belgium, Italy, Netherlands, and the United Kingdom. HTAs included a median of 15.5 studies (range 3 to 115).

[Figure omitted. See PDF.]

Table 1. Design of studies included in eligible health technology assessments.

https://doi.org/10.1371/journal.pone.0163032.t001

Primary study characteristics

The 265 primary studies were published between 1986 and 2014 by authors from 28 countries.

The majority of studies were published by authors in the United States (101/265, 38.1%). Among the 265 studies, 63 (23.8%) declared industry funding, 83 (31.3%) declared independent funding, and 119 (44.9%) offered no explicit conflict of interest statement or acknowledgement of funding.

Devices evaluated

The majority of devices were evaluated in very few studies (Table 2). For example, 26 of 47 (55.3%) TKA implants and 11 of 18 (61.1%) UKA implants were each evaluated in a single study.

[Figure omitted. See PDF.]

Table 2. Implants evaluated in eligible studies.

https://doi.org/10.1371/journal.pone.0163032.t002

Study design

Most studies evaluated TKA (198, 74.5%) (Table 3). Overall, the majority of studies were based on single cohorts (SC, 162, 61.1%), followed by comparative cohorts (CC, 62, 23.4%) and randomized controlled trials (41, 15.5%).

[Figure omitted. See PDF.]

Table 3. Number of patients evaluated in eligible studies.

https://doi.org/10.1371/journal.pone.0163032.t003

Participants

Across 265 studies, 59,217 patients were evaluated (Table 3). Notably, several studies failed to report the number of patients who received knee implants. Overall, most studies had 100 or fewer participants. By study design, 61.6% of patients were assessed in SC with a median (range) of 105.5 (14.0 to 4,393.0) patients in all TKA studies; 73.0 (10.0 to 881.0) patients in all UKA studies; and 97.0 (48.0 to 415.0) patients across all studies that evaluated both types of implants. This was followed by 31.7% of patients assessed in CC (TKA median 125.0, range 12.0 to 3,998.0; UKA median 88.0, range 28.0 to 447.0; both median 206.0, range 20.0 to 750.0); and 6.7% of patients assessed in RCTs (TKA median 92.0, range 20.0 to 390.0; UKA median 55.0, range 40.0 to 92.0; both median 39.0, range 22.0 to 56.0).

Follow-up period

Patients were followed for 2 years or less in a high proportion of studies for both UKA (SC 29.8%, CC 16.7%, RCT 25.0%) and TKA (25.0%, 31.4%, 48.6%) (Table 4)

[Figure omitted. See PDF.]

Table 4. Patient follow-up period in eligible studies by study design and knee system.

https://doi.org/10.1371/journal.pone.0163032.t004

Discussion

While hundreds of studies on knee arthroplasty devices were identified, there is little reliable data on the effectiveness and safety of most types of knee implants. Of 265 eligible primary studies, the findings of 70% of the primary studies that were industry- or undeclared sponsorship should be interpreted with some caution. The quality of evidence in primary studies was limited. Most studies were based on uncontrolled single cohorts, enrolled fewer than 100 patients, and followed patients for 2 years or less. Furthermore, most devices were evaluated in only one study. If safety or effectiveness of devices is a key concern, decisions regarding the choice of medical devices appear to be largely unsupported by reliable evidence.

Similar findings were identified in other assessments of syntheses and of primary studies. Sharma et al. [20] assessed the methodological quality of 77 meta-analyses in joint arthroplasty. Among these5 (6%) had extensive flaws, 34 (44%) had major flaws, 30 (39%) had minor flaws, and 8 (10%) had minimal flaws; the quality of 14 meta-analyses based on TKA was not reported. Nieuwenhuijse et al. [21] conducted a systematic review to appraise the evidence base for orthopedic devices including high flexion TKA. Among 56 studies describing 52 cohorts, study quality was judged to be low or moderate in over 60% of 56 studies describing 52 cohorts. However, our review was a more comprehensive assessment of the quality of primary studies on knee implants than either of these studies. Our meta-review included fewer syntheses of knee arthroplasty than the Sharma et al. study [20], but it included more detail about the quality of the primary studies. Our meta-review also included many more primary studies than the Nieuwenhuijse et al. review [21].

Our study has several strengths. We used rigorous meta-review methodology, and applied stringent eligibility criteria to retrieve the highest quality of evidence available. We may not have identified all eligible studies based on the search strategy employed, and because registry studies and non-English language studies were excluded. Notably, there was little overlap of primary studies across HTAs, in part due to the fact that the HTAs differed in the span of years they covered. Primary studies varied in the consistency and completeness of information they reported so it was difficult to extract and summarize data. Given that included studies were published as early as 1986, some of the devices evaluated in included studies may no longer be used.

The Balliol Collaboration issued recommendations for improving the evidence base for surgical innovations [22]. However, in the current market approval process, uncontrolled clinical studies may suffice as evidence for device effectiveness and safety, so there is little incentive for manufacturers to undertake additional or rigorous studies of a more costly nature [23]. Furthermore, many medical devices are not marketed for long before they are replaced by newer versions, and therefore fail to undergo sufficient, long-term evaluation [24]. Given the rapid rate of new medical device development and marketing [25], and tensions between system-level funding policies and organizational purchasing decisions [26], future research should investigate how to generate, synthesize and share evidence on the safety and effectiveness of medical devices in a manner that balances innovation and safety. Others have suggested that improved regulation (pre-market) and professional society oversight (post-market) strategies are both needed to optimize patient safety [21].

Although users of health technologies are expected to use evidence to guide decisions about the use of medical devices, our study of knee arthroplasty implants—among the most commonly used implantable devices in major surgical procedures—suggests that little high-quality evidence actually exists. Our study raises serious questions about the nature of clinical evidence supporting the safety and effectiveness of implants used for knee arthroplasty

Supporting Information

[Figure omitted. See PDF.]

S1 Checklist. PRISMA Checklist.

https://doi.org/10.1371/journal.pone.0163032.s001

(DOC)

S1 Table. Medline search strategy.

https://doi.org/10.1371/journal.pone.0163032.s002

(DOCX)

S2 Table. Web sites searched for HTAs.

https://doi.org/10.1371/journal.pone.0163032.s003

(DOCX)

S3 Table. AMSTAR Scoring.

https://doi.org/10.1371/journal.pone.0163032.s004

(DOCX)

S4 Table. Data extracted from primary studies and references.

https://doi.org/10.1371/journal.pone.0163032.s005

(DOCX)

Acknowledgments

We thank Jeremy Y. Ng for his assistance with data extraction and analysis.

Author Contributions

1. Conceptualization: ARG DRU.

2. Data curation: ARG JT.

3. Formal analysis: ARG JT CP.

4. Funding acquisition: ARG DRU.

5. Investigation: ARG AD PL SR PT AE CB JT CP DRU.

6. Methodology: ARG AD PL SR PT AE CB JT CP DRU.

7. Project administration: ARG JT.

8. Resources: ARG DRU.

9. Supervision: ARG DRU.

10. Validation: ARG AD PL SR PT AE CB JT CP DRU.

11. Visualization: ARG AD PL SR PT AE CB JT CP DRU.

12. Writing – original draft: ARG AD PL SR PT AE CB JT CP DRU.

13. Writing – review & editing: ARG AD PL SR PT AE CB JT CP DRU.

Citation: Gagliardi AR, Ducey A, Lehoux P, Ross S, Trbovich P, Easty A, et al. (2016) Meta-Review of the Quantity and Quality of Evidence for Knee Arthroplasty Devices. PLoS ONE 11(10): e0163032. https://doi.org/10.1371/journal.pone.0163032

References

1. Kramer DB, Mallis E, Zuckerman BD, Zimmerman BA, Maisel WH. Premarket clinical evaluation of novel cardiovascular devices: quality analysis of premarket clinical studies submitted to the Food and Drug Administration 2000–2007. Am J Ther. 2010;17:2–7. pmid:20038828

2. Rathi VK, Krumholz HM, Masoudi FA, Ross JSS. Characteristics of clinical studies conducted over the total product life cycle of high-risk therapeutic medical devices receiving FDA premarket approval in 2010 and 2011. JAMA. 2015;314:604–612. pmid:26262798

3. Langton DJ, Sidaginamale RP, Avery P, Waller S, Tank G, Lord J et al. Retrospective cohort study of the performance of the Pinnacle metal on metal (MoM) total hip replacement: a single-centre investigation in combination with the findings of a national retrieval centre. BMJ Open. 2016;6:e007847. pmid:27130159

4. Skou ST, Roos EM, Laursen MB, Rathleff MS, Arendt-Nielsen L, Simonsen O et al. A randomized, controlled trial of total knee replacement. NEJM. 2015;373:1597–1606. pmid:26488691

5. Wilson NA, Schneller ES, Montgomery K, Bozic KJ. Hip and knee implants: Current trends and policy considerations. Health Affairs. 2008;27:1587–1598. pmid:18997215

6. World Health Organization. WHO Health technology assessment of medical devices. Geneva: WHO Press, 2011. Available: http://apps.who.int/iris/bitstream/10665/44564/1/9789241501361_eng.pdf).

7. Moher D, Liberati A, Tetzlaff J, Altman DG and the PRISMA Group. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. PLoS Med. 2009;6:e1000097. pmid:19621072

8. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al.External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR). PLoS ONE. 2007;2: e1350. pmid:18159233

9. Pabinger C, Berghold A, Boehler N, Labek G. Revision rates after knee replacement. Cumulative results from worldwide clinical studies versus joint registers. Osteoarthritis Cartilage. 2013;21:263–268. pmid:23220555

10. Pabinger C, Lumenta DB, Cupak D, Berghold A, Boehler N, Labek G. Quality of outcome data in knee arthroplasty. Acta Orthop. 2015;86:58–62. pmid:25191934

11. Hofstede SN, Nouta KA, Jacobs W, van Hooff ML, Wymenga AB, Pijls BG, et al. Mobile bearing vs fixed bearing prostheses for posterior cruciate retaining total knee arthroplasty for postoperative functional status in patients with osteoarthritis and rheumatoid arthritis. Cochrane Database Syst Rev. 2015;2:CD003130. pmid:25650566

12. Kim SJ, Postigo R, Koo S, Kim JH. Causes of revision following Oxford phase 3 unicompartmental knee arthroplasty. Knee Surg Sports Traumatol Arthrosc. 2014;22:1895–1901. pmid:23996106

13. CADTH. Knee implant systems: A review of comparative clinical effectiveness. Ottawa, Canada: Canadian Association for Drugs and Technology in Health; 2013.

14. Smith TO, Hing CB, Davies L, Donell ST. Fixed versus mobile bearing unicompartmental knee replacement: A meta-analysis. Orthop Traumatol Surg Res. 2009;95:599–605. pmid:19942491

15. Cerbo M, Fella D, Jefferson T, Migliore A, Paone S, Perrini MR, Velardi L. Prostheses for primary total knee replacement in Italy. Rome, Italy: Agenas HTA; 2009.

16. Griffin T, Rowden N, Morgan D, Atkinson R, Woodruff P, Maddern G. Unicompartmental knee arthroplasty for unicompartmental osteoarthritis: a systematic review. Australian Safety and Efficacy Register of New Interventional Procedures—Surgical. Report No. 44. Adelaide, South Australia: The Royal Australasian College of Surgeons; 2005.

17. Medical Advisory Secretariat. Total knee replacement: an evidence-based analysis. Toronto, Canada: Ontario Ministry of Health and Long Term Care; 2005.

18. Ethgen O, Bruyère O, Richy F, Dardennes C, Reginster JY. Health-related quality of life in total hip and total knee arthroplasty. A qualitative and systematic review of the literature. J Bone Joint Surg Am. 2004;86-A:963–974. pmid:15118039

19. Kane RL, Saleh KJ, Wilt TJ, Bershadsky B, Cross WW III, MacDonald RM, Rutks I. Total knee replacement. Technology Assessment No. 86. AHRQ Publication No. 04-E006-2. Rockville, MD: Agency for Healthcare Research and Quality; 2003.

20. Sharma R, Vannabouathong C, Bains S, Marshall A, MacDonald SJ, Parvizi J, Bhandari M. Meta-analyses in joint arthroplasty: a review of quantity, quality, and impact. J Bone Joint Surg. 2011;93:2304–2309. pmid:22258777

21. Nieuwenhuijse MJ, Nelissen RG, Schoones JW, Sedrakyan A. Appraisal of evidence base for introduction of new implants in hip and knee replacement: a systematic review of five widely used device technologies. BMJ. 2014;349:g5133 pmid:25208953

22. McCulloch P, Altman DG, Campbell WB, Flum DR, Glasziou P, Marshall JC, et al. No surgical innovation without evaluation: the IDEAL recommendations. Lancet. 2009;374:1105–1112. pmid:19782876

23. Maisel WH. Medical device regulation: An introduction for the practicing physician. Ann Intern Med. 2004;140:296–302. pmid:14970153

24. Ross S, Robert M, Ducey A. The short life cycle of a surgical device. Literature analysis using McKinlay’s 7-stage model. Health Pol Technol. 2015;4:168–188.

25. Lexoux P, Williams-Jones B, Miller F, Urbach D, Taillez S. What leads to better health care innovation? Arguments for an integrated policy-oriented research agenda. J Health Serv Res Policy. 2008;13:251–4. pmid:18806185

26. Danjoux NM, Martin DK, Lehoux PN, Harnish JL, Shaul RZ, Bernstein M, Urbach DR. Adoption of an innovation to repair aortic aneurysms at a Canadian hospital: a qualitative case study and evaluation. BMC Health Serv Res. 2007;7:182. pmid:18005409