Health Services/Policy
Policy Exposure Bias: Simulating Information Bias in Legal Epidemiology Research Alina Schnake-Mahl* Alina Schnake-Mahl Gianni Anfuso Ellicott Matthay
Background: Epidemiologists are increasingly investigating the causal effects of social policies on health and health disparities. Previous work has identified potential sources of bias in policy analysis, including policy co-occurrence, time-varying confounders, statistical model misspecification, and limited power. However, less research has assessed the likely magnitude or implications of information bias arising from errors in measuring policy exposures. We use an existing systematic sample of social policy datasets developed by Matthay et al to quantify how much bias in causal effect estimates is likely to be introduced by various theoretical levels and sources of coding errors or date mismeasurement in policy variables.
Methods: Using simulations, we measure variation in the level of bias depending on whether mismeasurement is in the index policy, related policies that are controlled as confounders, or both. We also explore how the magnitude of bias varies when the measurement error is differential versus nondifferential with respect to confounding policies. We report absolute bias, 95% confidence interval coverage, root mean square error (RMSE), and variance for each metric. To complement and ground these simulations, we additionally review the 19 U.S. policy datasets from Matthay et al, and cross-validate the policy data in each database with the original legislative text to identify errors, and potential reasons errors emerged.
Results: In preliminary analyses, absolute bias ranged from 0.1-1.6 depending on the type of information bias. Six of 19 databases incorrectly measured dates, with the percent of incorrect policy dates ranging from 2-40%; differences between the incorrect and correct dates ranged from 2 to 794 days. The most common reasons for incorrect dates included legal battles about legislation, repealed policies, and multiple versions of a bill before the final bill that passed.
Conclusions: Coding errors and date mismeasurement of policy exposure can produce substantial levels of bias. Rigorous application of legal epidemiology and policy surveillance methods can help address these sources of bias. Epidemiologists may benefit from additional training in these areas or partnering with policy experts to mitigate potential information bias.