The small trial problem

Abstract

Background

Many randomized trials that aim to assess new or commonly used medical or surgical interventions have been so small that the validity of conclusions becomes questionable.

Methods

We illustrate the small trial problem using the power calculation of five Cochrane-reviewed studies that compared vertebroplasty versus placebo interventions. We discuss some of the reasons why the statistical admonition not to dichotomize continuous variables may not apply to the calculation of the number of patients required for trials to be meaningful.

Results

Placebo–controlled vertebroplasty trials planned to recruit between 23 and 71 patients per group. Four of five studies used the standardized mean difference of a continuous pain variable (centimeters on the visual analog scale (VAS)) to plan implausibly small trials. What is needed is not a mean effect at the population level but a measure of efficacy at the patient level. Clinical practice concerns the care of individual patients that vary in many more respects than the variation around the mean of a single selected variable. The inference from trial to practice concerns the frequency of success of the experimental intervention performed one patient at a time. A comparison of the proportions of patients reaching a certain threshold is a more meaningful method that appropriately requires larger trials.

Conclusion

Most placebo-controlled vertebroplasty trials used comparisons of means of a continuous variable and were consequently very small. Randomized trials should instead be large enough to account for the diversity of future patients and practices. They should offer an evaluation of a clinically meaningful number of interventions performed in various contexts. Implications of this principle are not specific to placebo-controlled surgical trials. Trials designed to inform practice require a per-patient comparison of outcomes and the size of the trial should be planned accordingly.