Assessment of Methodologic Quality of Randomized Trials of Interventional Techniques: Development of an Interventional Pain Management Specific Instrument

Abstract

BACKGROUND: A major component of a systematic review is an assessment of the methodological quality and bias of randomized trials. The most commonly utilized methodological quality assessment and bias assessment for randomized trials is by the Cochrane Review Group. While this is not a “gold standard,” it is an indication of the current state-of-the-art review methodology. There is, however, no specific instrument to assess the methodological quality of manuscripts published for interventional techniques.

OBJECTIVES: Our objective was to develop an instrument specifically for interventional pain management, to assess the methodological quality of randomized trials of interventional techniques.

METHODS: Item generation for the instrument was based on a definition of quality, to the extent to which the design and conduct of the trial were congruent with the objectives of the trial. Applicability was defined as the extent to which the trial produced procedures could be applied with contemporary interventional pain management techniques. Multiple items based on Cochrane review criteria were utilized along with specific requirements for interventional techniques.

RESULTS: A total of 22 items were developed which formed IPM-QRB or Interventional Pain Management Techniques – Quality Appraisal of Reliability and Risk of Bias Assessment tool. This included 9 of the 12 items from the Cochrane review criteria with deletion of some items that were repetitive or duplicate, and the addition of 13 new items.

The results were compared for inter-rater reliability of Cochrane review criteria and IPM-QRB, and inter-instrument reliability.

The assessment was performed in multiple stages with an initial learning curve. The final assessment was for 4 randomized controlled trials (RCTs) utilizing both Cochrane review criteria and IPM-QRB criteria. The inter-rater agreement for Cochrane review criteria with overall intra-class correlation coefficient was 0.407 compared to an intra-class correlation coefficient of 0.833 for IPM-QRB criteria. The inter-rater agreement was superior for IPM-QRB criteria compared to Cochrane review criteria despite twice the items of Cochrane review criteria as IPM-QRB criteria with the detailed nature of assessment.

LIMITATIONS: Limited validity or accuracy assessment of the instrument and the large number of items to be scored.

CONCLUSION: We have developed a new comprehensive instrument to assess the methodological quality of randomized trials of interventional techniques. This instrument is superior to Cochrane review methodology criteria in that it provides more extensive and specific information for interventional techniques that will be useful in assessing the methodologic quality and bias of interventional techniques.