Economic and statistical measures of forecast accuracy

Headnote

ABSTRACT

This paper argues in favour of a closer link between the decision and the forecast evaluation problems. Although the idea of using decision theory for forecast evaluation appears early in the dynamic stochastic programming literature, and has continued to be used with meteorological forecasts, it is hardly mentioned in standard academic texts on economic forecasting. Some of the main issues involved are illustrated in the context of a two!state, two! action decision problem as well as in a more general setting. Relationships between statistical and economic methods of forecast evaluation are dis! cussed and links between the Kuipers score used as a measure of forecast accuracy in the meteorology literature and the market timing tests used in -nance are established. An empirical application to the problem of stock market predictability is also provided, and the conditions under which such predictability could be explained in the presence of transaction costs are discussed. Copyright (c) 1999 John Wiley & Sons, Ltd.

KEY WORDS decision theory^ forecast evaluation^ probabilistic forecasts^ economic and statistical measures of forecast accuracy^ stock market predictability

INTRODUCTION

In the real, non!academic world forecasts are made for a purpose and the relevant purpose in economics is to help decision makers improve their decisions. It follows that the correct way to evaluate forecasts is to consider and compare the realized values of di}erent decisions made from using alternative sets of forecasts. In the academic literature there are frequent mentions of this viewpoint but few attempts to carry forward into a practical example. Forecast evaluation, when considered at all, is in terms of statistical accuracy measures of point forecasts and standard forecasting textbooks do not discuss the decision!making aspects. See, for example, Box and Jenkins "0869#, Granger and Newbold "0875#, and Clements and Hendry "0887, 0888#. An early discussion of the usefulness of the decision approach is in a book by Theil "0859# whose Sections 7.3 and 7.4 are similar in spirit to our discussion, although quite di}erent in technique. Another important early reference is White "0855# who considers decision theory for forecast evaluation in the dynamic stochastic programming literature.

The forecasting area that is most advanced in using decision theory for evaluation is meteor! ology and in later sections we will review some of...