Causal Inference, Shaolin Style: "Mastering 'Metrics"*

Abstract

[...]the authors discuss a paper by Clark and Martorell (2014) that employs a clever fuzzy RDD strategy using last-chance exams in Texas. [...]the approachable style of the book can be stimulating to students, so it is a good recommendation for those teaching quantitative methods at undergraduate level.

Causal Inference, Shaolin Style: "Mastering 'Metrics"* (Angrist, Joshua D. and Pischke, Jörn-Steffen. Mastering 'Metrics: The Path From Cause to Effect. Princeton University Press, 2014)

The field of causal inference has seen a remarkable development in the last few years. While social scientists have always devoted considerable effort to understand causal effects, their quest for the perfect identification strategy received a new impulse after the publication of Donald Rubin's causal model in the late 1970s (HOLLAND, 1986; MORGAN and WINSHIP, 2007). Although the "potential outcomes revolution" is still in its infancy, a myriad of statistical methods have been designed to help researchers untangle true relationships from spurious effects (IMBENS and RUBIN, 2015). However, such methods already have an established place in cutting-edge publications. They remain relatively unknown to undergraduate students of economics and to academics in other areas.

Joshua D. Angrist and Jörn-Steffen Pischke try to fill this gap with their new book, Mastering 'Metrics: The Path From Cause to Effect (2014). Angrist and Pischke are household names in applied econometrics-'metrics in economics parlance-not only because of their many influential papers on labor markets and education but mainly for their companion to the aspiring econometrician, Mostly Harmless Econometrics (2008).

But, in contrast with its famous sibling, Mastering 'Metrics was not written with a Ph.D. candidate in mind. 'Metrics is very light on mathematics and its many Kung Fu-themed jokes create a conversational tone, which is lacking in more formal texts. This is perhaps the first econometrics book that can be read without much effort, making it suitable to those meeting the discipline for the first time or to scholars with little quantitative background.

'Metrics is focused exclusively on what the authors call "the furious five", a set of statistical techniques to assess causal relationships from observational data. The methods are: randomized trials, regression with matching, instrumental variables (IV)/two-stage least squares (2SLS), regression discontinuity design, and differences in differences (DID). Angrist and Pischke (2014) dedicate one chapter to each technique, and all of them follow the same structure. They start with one or two empirical examples of how the method is used, continue with a theoretical discussion, and conclude with a brief appendix. The appendices include recommended readings, basic mathematical notation, and a short note on the researcher who developed the technique. Each chapter is self-sufficient and can be read independently.

The book begins with an exposition of potential outcomes and experimental random assignment, the foundations of Rubin's causal model of inference (HOLLAND, 1986; RUBIN, 1974, 1978). Then the authors ask a puzzling question: does health insurance make people healthier? The answer is not as trivial as it seems. A simple comparison between those who are insured and those who are not shows that the first group tends to enjoy considerably better health standards than the latter. The tricky part is that those groups cannot be directly compared because of selection bias: people who can afford private health insurance are usually richer than average, so they exercise more, have a well-balanced diet and other things. A reliable inference strategy should be able to mitigate such unobservable differences and estimate the true causal effect. Ideally, a researcher would measure the same individual in two hypothetical situations, when he or she has health insurance and when he or she does not, and then take the difference between both "states of the world". However, it is evident, that only one of these outcomes is actually observable. This is what Holland (1986) calls "the fundamental problem of causal inference": it is impossible to observe an individual with two characteristics at the same time. A reasonable solution to this problem is to have two randomly chosen groups of people-one that receives the treatment and another that does not- average both of them and calculate the difference between the results. If the samples are large enough and the treatment condition has been randomly assigned, all biases will "average out" and the groups can thus be meaningfully compared. After presenting the assumptions required by the experimental design (such as ceteris paribus and non-interference between units), the authors mention that the state of Oregon involuntarily conducted a true randomized experiment when it decided to offer health insurance through lottery to about 75,000 people. Results show that the health insurance coverage did help people to get more medical treatment, but it had no effect when it comes to blood pressure or other medical conditions.

Chapter two deals with regression methods. The motivating example is the impact of schooling on wages. But how does one estimate the effect correctly when people have gone to colleges as different as the University of Massachusetts and Harvard? Angrist and Pischke (2014) take that question as a starting point to offer a detailed explanation of a common problem in regression analysis: omitted variable bias (OVB). This type of bias occurs when the researcher does not include an important independent variable in the model he wants to estimate, thus biasing the coefficients of the other factors. As the authors note, "regression is a way to make other things equal, but equality is generated only for variables that are included as controls on the right-side of the model" (ANGRIST and PISCHKE, 2014, p. 67). In the school example above, the authors suggest that family size or income may have determined if a student would go to college A or B, and therefore, influenced his future income. Without such controls, there is no ceteris paribus, the assumption that "all other things should be equal". Curiously, the authors do not present any modern matching technique to pre-process the data before running a regression model (e.g., HO et al., 2007; ROSENBAUM and RUBIN, 1985), although this is a widespread practice amongst quantitative scholars (STUART, 2010). A short mention would have solved the issue.

The next chapter introduces the reader to IV and the 2SLS regression framework. IV serve a useful purpose in applied research, enabling scholars to estimate unbiased coefficients and assess causality when randomization is unfeasible. A reliable instrument should possess three distinct characteristics: 1) it has a causal effect on the independent variable one is trying to evaluate (the "treatment"); 2) it is unrelated to omitted variables the researcher is willing to control for; 3) it affects the dependent variable though a single channel, the treatment. If those conditions are met, the estimation can be considered "as good as if it was obtained with an experiment" (ANGRIST and PISCHKE, 2014, p. 106). In turn, 2SLS gives flexibility to IV estimations by allowing the analyst to add other independent variables to control for possibly omitted factors in his or her model. In this regard, 2SLS can substantially increase the accuracy of the estimations and allow for continuous, dichotomous, and ordinal instruments to be included in a regression model. Two examples are used to illustrate the method: the first one is on the impact of charter school on wages (taking a lottery as an instrument), and the second one explains the influence of family size and wealth using the random incidence of twins as an IV.

The authors proceed to a discussion on regression discontinuity design (RDD). This method is adequate to analyze natural experiments where an intervention caused a (sharp or fuzzy) cutoff point in the independent variable of interest. In their book example, Angrist and Pischke (2014, p. 150) employ a RDD to evaluate the causal effect of legal access to alcohol on death rates. There is a clear threshold that indicates legal drinking age in the United States (21 years old), and the authors use that jump to address their question. The theory behind RDD suggests that if one takes the cases that lie closely to the discontinuity, all of them have very similar characteristics apart from the causal effect one is trying to estimate. Thus, the difference between the death rate of youngsters who are about to turn 21 and those who have just passed that date are due to legal alcohol consumption. The results, for those who are curious, are that there is indeed a huge spike in death rates for youngsters who are 21 years old. The authors also discuss fuzzy RDD, i.e., when the probability of an event happening increases as an individual approaches the cutoff. In this sense, RDD is similar to an IV where the discontinuity becomes an instrumental variable for treatment status. This concept is not very intuitive and it is difficult to explain without resorting to mathematics, but the authors do a good job by presenting the intuition behind sharp and fuzzy RDD with graphs. Although one sometimes cannot be sure if there is a jump by merely looking at covariate plots, it is probably the best way to approach the problem.

The last methodological chapter in the book analyzes DID. This method is based upon the common trends assumption, which states that, "sometimes, treatment and control outcomes move in parallel in the absence of treatment" (ANGRIST and PISCHKE, 2014, p. 178). When this is the case-and it is possible to test it visually and mathematically-one can attribute the shift in the expected trend to a causal effect, estimated by subtracting the average gain over time in the control group from the gain over time in the exposed group (IMBENS and WOOLDRIDGE, 2009, p. 67). Such a procedure removes eventual selection biases from the analysis. A study of the impact of different monetary policies on two of the twelve US Federal Reserve districts in 1929 is the authors' case study of choice. The DID results indicate that looser monetary policies led to less trouble in the financial sector, with fewer banks going bankrupt in posterior years. The effect of monetary policy can be estimated with a dummy variable that captures the state that had the treatment, a time indicator for post-treatment periods, and an interaction term composed by both of them. The coefficient resulting from that multiplication is the DID causal effect (ANGRIST and PISCHKE, 2014, p. 186). The framework is flexible enough to include multiple years and comparison groups, as the authors note. Angrist and Pischke analyze a case of multiple legal drinking age being implemented in different states and estimate their impact on death rates, the same question presented in the earlier chapter on RDD. After controlling for state and time effects, the authors can obtain unbiased results with DID.

Mastering 'Metrics concludes with a thorough investigation of a hot topic in labor and education economics, the effect of schooling on earnings. The authors present a myriad of studies that tackle the question using all methods shown in the book apart from experiments. There has never been a truly randomized controlled trial on returns to schooling, hence, no example of that technique is provided. The chapter begins with Jacob Mincer's (1974) and Zvi Griliches' (1977) regressions on returns to schooling, presenting the pros and cons of those pioneering studies. Angrist and Pisckhe (2014) then move to an evaluation of the work of Ashenfelter and Krueger (1994) and Ashenfelter and Rouse (1998) using twins as an IV for the difference in education. The results seem to confirm Mincer's regressions. Next, the authors show how DID and IV can be combined. They present two papers published by Acemoglu and Angrist (2001) and Angrist and Krueger (1991) on the effects of compulsory attendance laws on returns to schooling. By using both methods one can ameliorate OVB issues. Finally, the authors discuss a paper by Clark and Martorell (2014) that employs a clever fuzzy RDD strategy using last-chance exams in Texas. The book ends with a short comment on measurement errors and notes regarding the data sources.

How can a political scientist benefit from reading Mastering 'Metrics? As the discipline moves towards a more rigorous approach to causality, the lessons offered by Angrist and Pischke (2014) are well timed. Moreover, the approachable style of the book can be stimulating to students, so it is a good recommendation for those teaching quantitative methods at undergraduate level.

Nevertheless, the book has its shortcomings, too. 'Metrics has a much more limited scope than traditional textbooks such as Stock and Watson (2003) or Wooldridge (2012). Angrist and Pischke's (2014) discussion of causality is solely based upon the linear model, and all techniques presented in the book are applied only to ordinary least squares. In this regard, the authors do not provide any hint on how to employ their causality toolbox to dichotomous, ordinal, count, or nominal variables. Even time series are absent from the book. Those looking for an introduction to those topics should therefore, refer to one of the volumes mentioned above. Furthermore, 'Metrics does not include any exercises, and although the authors claim that additional explanatory material will soon be available online1, so far students cannot practice the concepts they have learnt in the book.

Despite its flaws, Mastering 'Metrics provides an enriching and even entertaining introduction to five important econometrics methods. Its discussion of causality is very clear, the examples are relevant and the appendices present the mathematical notation readers may need to follow more advanced texts. Although the book alone will not make one a Shaolin master of econometrics, it will lead the hopeful apprentice onto the righteous path. And this is not a small feat.

Revised by Cabo Verde

Footnote

(*) http://dx.doi.org/10.1590/1981-38212015000300026

1 See: http://masteringmetrics.com/resources/ . Access on February 02, 2015.

References

References

ACEMOGLU, Daron and ANGRIST, Joshua (2001), How large are human-capital externalities? Evidence from compulsory-schooling laws. In: NBER Macroeconomics Annual 2000. Edited by Ben Bernanke and Kenneth Rogoff. Cambridge: MIT Press.

ANGRIST, Joshua and KRUEGER, Alan (1991), Does compulsory school attendance affect schooling and earnings? Quarterly Journal of Economics. Vol. 106, No 04, pp. 979-1014.

ANGRIST, Joshua and PISCHKE, Jörn-Steffen (2008), Mostly harmless econometrics: An empiricist's companion. Princeton: Princeton University Press. 392pp.

ANGRIST, Joshua and PISCHKE, Jörn-Steffen (2014), Mastering 'metrics: The path from cause to effect. Princeton: Princeton University Press. 304pp.

ASHENFELTER, Orley and KRUEGER, Alan (1994), Estimates of the economic return to schooling from a new sample of twins. The American Economic Review. Vol. 85, No 05, pp. 1157-1173.

ASHENFELTER, Orley and ROUSE, Cecilia (1998), Income, schooling, and ability: Evidence from a new sample of identical twins. The Quarterly Journal of Economics. Vol. 113, No 01, pp. 253-284.

CLARK, Damon and MARTORELL, Paco (2014), The signaling value of a high school diploma. Journal of Political Economy. Vol. 122, No 02, pp. 282-318.

GRILICHES, Zvi (1977), Estimating the returns to schooling: Some econometric problems. Econometrica: Journal of the Econometric Society. Vol. 45, No 01, pp. 01- 22.

HO, Daniel; IMAI, Kosuke; KING, Gary and STUART, Elizabeth (2007), Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Political analysis. Vol. 15, No 03, 199-236.

HOLLAND, Paul (1986), Statistics and causal inference. Journal of the American Statistical Association. Vol. 81, No 396, pp. 945-960.

IMBENS, Guido and RUBIN, Donald (2015), Causal inference in statistics, and in the social and biomedical sciences. Cambridge: Cambridge University Press. 592pp.

IMBENS, Guido and WOOLDRIDGE, Jeffrey (2009), Recent developments in the econometrics of program evaluation. Journal of Economic Literature. Vol. 47, No 01, pp. 05-86.

MINCER, Jacob (1974), Schooling, Experience, and Earnings. Cambridge: National Bureau of Economic Research. 149pp.

MORGAN, Stephen and WINSHIP, Christopher (2007), Counterfactuals and causal inference: Methods and principles for social research. Cambridge: Cambridge University Press. 328pp.

ROSENBAUM, Paul and RUBIN, Donald (1985), Constructing a control group using multivariate matched sampling methods that incorporate the propensity score. The American Statistician. Vol. 39, No 01, pp. 33-38.

RUBIN, Donald (1974), Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of educational Psychology. Vol. 66, No 05, pp. 688- 701.

RUBIN, Donald (1978), Bayesian inference for causal effects: The role of randomization. The Annals of Statistics. Vol. 06, No 01, pp. 34-58.

STOCK, James and WATSON, Mark (2003), Introduction to econometrics. Boston: Addison Wesley. 379pp.

STUART, Elizabeth (2010), Matching methods for causal inference: A review and a look forward. Statistical science. Vol. 25, No 01, pp. 01-29.

WOOLDRIDGE, Jeffrey (2012), Introductory econometrics: A modern approach. Boston: Cengage Learning. 912pp.