Content area
Full Text
Contents
- Abstract
- Interpreting Evidential Value and Lack Thereof
- P-Curve’s Many Uses
- P-Curve’s Shape
- Heterogeneity and p-Curve
- Statistical Inference With p-Curve
- Does a Set of Studies Contain Evidential Value? Testing for Right-Skew
- Does a Set of Studies Lack Evidential Value? Testing for Power of 33%
- Logic underlying the test
- Implementation of the test
- A Demonstration
- Selecting Studies
- Selecting p Values
- Step 1. Identify Researchers’ Stated Hypothesis and Study Design (Columns 1 and 2)
- Step 2. Identify the Statistical Result Testing the Stated Hypothesis (Column 3)
- Step 3. Report the Statistical Result(s) of Interest (Column 4)
- Step 4. Recompute Precise p Values Based on Reported Test Statistics (Column 5)
- Step 5. Report Robustness p Values (Column 6)
- P-Curving Findings That Do Not Test the Researcher’s Stated Hypothesis
- Selecting p Values for Specific Study Designs
- How Often Does p-Curve Get It Wrong?
- Cherry-Picking p-Curves
- How concerned should we be with cherry-picking? Not too much
- How to prevent cherry-picking? Disclosure
- P-Curve Versus Other Methods
- Limitations
- Why Focus on p Values?
- Conclusions
Figures and Tables
Abstract
Because scientists tend to report only studies (publication bias) or analyses (p-hacking) that “work,” readers must ask, “Are these effects true, or do they merely reflect selective reporting?” We introduce p-curve as a way to answer this question. P-curve is the distribution of statistically significant p values for a set of studies (ps < .05). Because only true effects are expected to generate right-skewed p-curves—containing more low (.01s) than high (.04s) significant p values—only right-skewed p-curves are diagnostic of evidential value. By telling us whether we can rule out selective reporting as the sole explanation for a set of findings, p-curve offers a solution to the age-old inferential problems caused by file-drawers of failed studies and analyses.
Scientists tend to publish studies that “work” and to place in the file-drawer those that do not (Rosenthal, 1979). As a result, published evidence is unrepresentative of reality (Ioannidis, 2008; Pashler & Harris, 2012). This is especially problematic when researchers investigate nonexistent effects, as journals tend to...