Content area
Abstract
This dissertation consists of three essays that contribute to the political science scholarship on meta-analysis and measurement validity.
In the first paper, I develop a multi-method approach to meta-analysis in political science. Meta-analysis is a sophisticated and essential tool for cumulative knowledge production. However, traditional approaches to meta-analysis systematically exclude qualitative scholarship. This project proposes a systematic framework to deal with this shortcoming of meta-analysis: Bayesian Integrative Meta-Analysis (BIMA). Drawing on existing literature in Bayesian elicitation, I show how information from qualitative manuscripts can be systematically converted into Bayesian prior distribution information via a process I call conversion elicitation. This converted qualitative information can be combined and integrated into a Bayesian meta-analysis. The resulting posterior distribution of the standardized effect size results from a qualitatively informed prior and a likelihood distribution composed of the effect sizes from quantitative studies. I explicate the framework for BIMA using a toy case with simulated data and an applied example related to the effect of competitive wages on bureaucratic corruption. In addition to providing a framework for meta-analysis of diverse evidence types, this research contributes to the methodological scholarship on multi-method research tools. Strategies for mixed methods political science research have primarily focused on combining methods for causal research. This contribution presents an avenue for mixed methods approaches for other objectives, such as knowledge synthesis.
In the second paper, I develop a framework for the informed use of proxy variables. Throughout the social sciences, practical challenges to measurement may frustrate scholars' attempts to measure their concepts of interest directly. In these cases of costly concepts, researchers may be compelled to use proxy variables that substitute values for the true concept measurement instead, often only assuming that the chosen proxy is good enough. In this paper, I propose an intermediate-range solution for discovering and reporting the potential measurement divergence between misaligned proxies and the true measures they intend to capture. I suggest obtaining a validation sample for which true measurements are obtained for the concept of interest is always preferable to assuming that a chosen proxy is sufficient. I outline three useful metrics to estimate, report, and potentially incorporate into additional statistical inference strategies. Furthermore, I use a simulation study to demonstrate which sampling procedures are most effective for obtaining these quantities. For more complex population structures, random or traditional stratified samples are informative in estimating metrics related to proxy-costly concept divergence. However, other stratified sampling approaches that consider the potential likelihood of measurement divergence across a population can also be informative of measurement divergence. This guide to validating proxies and reporting measurement divergence provides scholars with a practical and straightforward tool to account for the limitations of measurement choices.
In the third paper, I develop a novel categorization of the distinct practical challenges that social scientists may encounter when attempting to measure concepts: resource constraints, ethical obligations, confidence in indicators, and unit divergence. I illustrate these practical challenges of concept measurement using various concepts throughout political science and discuss how these challenges may intersect with more theoretical considerations related to translating concepts into operational dimensions. I develop a cost-benefit roadmap for devising alternative concept measurements when researchers encounter insurmountable challenges to pursuing a preferred, theoretically-driven measure. Though this paper's primary contribution is to the scholarship on concept measurement, it also has implications for transparency in social science. The roadmap I develop provides scholars with a conceptual vocabulary to justify their concept measurement choices, allowing scholars to be more explicit about how different logistical costs of research contour methodological and design choices.
Details
