Content area
Full text
When media analysts see a candidate's lead in three successive tracking polls grow from 2 percent to 3 percent to 9 percent, they generally report that the current lead stands at 9 percent. That's a tempting story to tell, but it's unlikely to be true.
EACH ELECTION SEASON, pollsters conduct "tracking polls" to measure public opinion over time. Some seek to provide strategic information to political campaigns, while others aim to supply interesting facts to journalists, pundits and scholars. Whatever the objectives, producers and consumers of tracking poll data have an interest in maximizing the accuracy of the results.
The limiting factor, however, is money. Surveys are expensive. Much as one might like to obtain precise results by gathering a large number of interviews, the bigger the sample, the more daunting the price tag. Given this trade-off between cost and accuracy, most efforts to track opinion over time rely on surveys of between 200 and 1,000 respondents.
That's why you should be careful when analyzing day-to-day tracks yourself or when listening to such analyses from so-called experts. A well-conducted survey of 1,000 respondents is nothing to sneeze at, but it's not infallible either. Suppose we were tracking the percentage of the public that holds a favorable impression of the president, and suppose that unbeknownst to us, the true percentage is 55 percent.
Even under the best conditions the sampling error associated with our survey's assessment of the president's favorability rating is 1.6 percent, which means that under the usual assumptions, 95 percent of the surveys we might undertake will produce favorability ratings between 52 percent and 58 percent. So if one day the president's approval rating stands at 57 percent, and the next it falls to 53 percent, it is entirely possible that the apparent change in opinion is due to random sampling variability. Different surveys conducted in an identical manner may produce somewhat different results.
All this is common knowledge among pollsters. Yet the implications of sampling variability for the interpretation of polls are routinely ignored.
Consider, for example, the election year commentary of many political pundits and reporters. When media analysts see a candidate's lead in three successive tracking polls grow from 2 percent to 3 percent to 9 percent, they generally...