Content area
Full Text
Assessment of the effects of area-wide service or public health interventions, or the impact of technologies that are evolving over time, often involves non-randomised comparisons between populations in different places or measured at different times.
Although empirical studies comparing randomised and non-randomised controlled trial (NRCT) studies have shown mixed results, some finding that NRCT give biased estimates compared with randomised controlled trials and some that they do not, 1 - 8 all commentators agree that case-mix adjustment is an essential mark of quality in non-randomised comparisons evaluating interventions.
A recent study by Deeks et al . 9 has, however, found that NRCT using before and after or contemporary cohort designs are biased, and furthermore that case-mix adjustment is problematical, always increasing the variability of the estimated effect and rarely eliminating the bias. These findings are perhaps not surprising because in "one-dimensional" designs that compare before and after periods, or contemporary populations, there are likely to be some case-mix differences that affect outcome but have not been measured, often because they were not known about, and which have not therefore been taken into account. Ignoring this problem, and assuming that case-mix adjustment leads to unbiased comparisons has been termed "the case-mix fallacy". 10
More surprising was the finding by Deeks et al . 9 that case-mix adjustment in one-dimensional NRCT not only failed to eliminate all the bias but sometimes increased it, and this cannot have been caused by failure to adjust for unknown covariates. One possible cause of this problem is that the relationship between the case-mix variable and the outcome differs between the populations or time periods being compared. Ignoring these interaction effects in a case-mix adjustment model commits what might be termed the "constant risk fallacy".
THE CONSTANT RISK FALLACY: WHAT IS IT?
Consider, for example, comparing outcomes or health service utilisation between populations or time periods adjusting for different socioeconomic status patterns using car ownership or level of education attainment. Conventional risk adjustment models ignore the fact that car ownership and educational attainment may "mean" different things in different populations and at different times. In the United Kingdom, for example, the level of "risk" indicated by having a higher education may have changed as the numbers have quadrupled in the past 20...