Methods/Statistics
The dwindling yet lingering myths about use of p-values and confidence intervals in modern epidemiology Montana Kekaimalu Hunter* Montana Kekaimalu Hunter Anthony Russell Igor Burstyn George Maldonado
Background: Misinterpretation of statistical significance tests and confidence intervals has been a longstanding issue in epidemiology. Despite efforts by leading journals to discourage or ban such practices, the extent of misinterpretations in modern epidemiologic literature remains unclear.
Methods: We examined papers published in 2022 in three leading epidemiology journals (International Journal of Epidemiology, Epidemiology, and American Journal of Epidemiology) to assess the frequency and types of misinterpretations of p-values and confidence intervals. We randomly sampled 65 papers that assessed exposure-outcome relationships. Two authors independently reviewed the selected papers, cataloging misinterpretations according to guidelines published in 2016.
Results: Among the examined papers, 10 (almost one-seventh) contained misinterpretations of statistical significance. The most common error in seven papers was treating the p-value as a property of the phenomenon being studied rather than a characteristic of the test result. Three papers incorrectly interpreted 95% confidence intervals as having a 95% chance of containing the true effect size. Examples of misinterpretations included concluding no effect based on p>0.05, equating statistical significance with substantive importance, and using confidence intervals to judge “significance.” We also highlight some commendable uses of p-values.
Conclusions: While misinterpretations of statistical tests were not widespread in top epidemiology journals, nearly one-seventh of the papers reviewed contained such errors. These findings highlight the need for continued efforts to improve the understanding and reporting of statistics in epidemiology.