Infectious Disease
An evaluation of evaluations: reflections on the COVID-19 emergency in New York City Sharon K. Greene* Sharon Greene Shuo Feng Alyssa Bilinski
Background: Understanding the real-world impact of interventions to mitigate infectious disease is crucial for public health planning, policymaking, and building public trust. Observational methods for causal inference have become popular for assessing the effectiveness of policies instituted under non-emergency conditions. However, conducting evaluations during infectious disease emergencies presents unique challenges. We propose a framework for emergency policy evaluation, informed by past evaluations conducted during the COVID-19 emergency response in New York City (NYC).
Methods: We reviewed evaluations of policies involving the NYC Department of Health and Mental Hygiene related to COVID-19, documenting objectives, data sources, methods, and opportunities for expanding or improving future evaluations. We then characterized potential barriers to policy evaluation during future emergencies and outlined infrastructure and methodology to support real-time evaluation.
Results: We identified three key considerations for planning policy evaluations: 1) methods selection; 2) timely data on outcomes of interest, including for comparison groups unexposed to the policy; and 3) interpretation and implementation strategy (Table 1). During the COVID-19 public health emergency, mandatory individual-level reporting of immunizations for persons of all ages and of negative test results provided granular data that enabled rapid, high-quality evaluations. Future evaluations could be strengthened by applying recently developed methods that account for non-linearities in infectious disease transmission and by interrogating the sensitivity of results to different modeling choices.
Conclusion: Public health agencies should plan evaluations prospectively at the time of policy roll-out. Collecting data on comparison groups is critical for population-level evaluations, as is appropriately accounting for infection dynamics.
