Methods/Statistics
Which policy works and where? Estimation and inference for state level treatment effects using difference-in-differences Nichole Austin* Nichole Austin Sunny Karim Matt Webb Erin Strumpf
New difference-in-differences (DID) estimators have been developed to estimate treatment effects in the context of staggered policy adoption. Most estimate the average treatment effect on the treated (ATT) at the treatment timing group (g) and year (t) level, or the ATT(g,t). However, this overlooks heterogeneity within groups: for example, in an analysis of state-level abortion bans, policies implemented at the same time may vary in terms of travel restrictions or gestational age thresholds. Accordingly, researchers may be interested in treatment effects at the state level which map more directly to policy characteristics.
We demonstrate how to estimate state-by-year level treatment effects (ATT(s,t) terms) using two new DiD estimators: unpooled DID (UN-DID) and intersection DID (DID-INT). We propose methods for cluster robust inference for ATT, ATT(g,t) and ATT(s,t) terms using these estimators. The cluster jackknife and randomization inference are of broader interest, as they are valid in settings where the standard cluster robust variance estimator is not. We used Monte Carlo simulation to compare traditional DID to UN-DID/ DID-INT with and without cluster robust estimation.
The new methods recover the true ATT(s,t) values. First, all over-time differences are calculated by state. Within-state differences for controls are then aggregated. Comparing this difference with a single treated state’s difference yields an ATT(s,t). Conversely, if it is compared to the aggregated treated differences by timing group (g), we estimate an ATT(g,t). Our new estimators are unbiased and correctly sized in most settings (e.g., when the number of clusters is sufficient).
In policy evaluation, the parameter of interest is often the ATT(s,t), but the ATT(g,t) is typically reported. We introduce methods to estimate the ATT(s,t). Both estimates are important in understanding the impact of policy shifts that are heterogeneous in terms of policy timing and characteristics.