Background: Systematic social observation (SSO) can objectively measure changes in the social and built environment, but its application in evaluating the effects of overdose prevention programs (OPPs) on public disorder is limited.

Methods: We developed, implemented, and validated an SSO instrument in New York City (NYC) and Rhode Island (RI). Research assistants (RAs) assessed neighborhoods surrounding OPPs from 4/19/2023-6/22/2023 in NYC and 11/4/2024-12/20/2024 in RI. We used a novel randomization schedule to control for temporal fluctuations in public disorder. Among paired observations with simultaneous assessments by two RAs in NYC, we used Gwet’s AC1 to assess inter-rater reliability (IRR). Among independent observations by a single RA, we used exploratory factor analysis (EFA) and Bayesian multi-level item response theory (IRT) models to derive separate latent measures of public disorder for NYC and RI and externally validated the measures against local arrest data.

Results: Among 19 paired observations, IRR was suitable or near perfect (Gwet’s AC1 ≥60%) for 80% of categorical items (n=43/54). Separate EFA among independent observations from NYC (n=564) and RI (n=463) each identified three factors assessed with the instrument: drug litter, people, and built environment. IRT-based latent measures of public disorder (estimated from the three identified factors) exhibited positive spatial autocorrelation (NYC Moran’s I=0.393, P<0.001; RI Moran’s I=0.068, P<0.001). In NYC, item severity and discrimination were highest for the drug litter factor (Table). In RI, item parameters were relatively uniform across factors. The NYC latent measure was associated with total arrests (incidence rate ratio=1.42, 95% confidence interval [CI]=1.04-1.95) and drug-related arrests (incidence rate ratio=1.60, 95% CI=1.10-2.32). The latent measure was not associated with total arrests or any arrest subtype in RI.

Conclusion: The SSO instrument had strong psychometric properties. The latent measures of public disorder may be useful outcomes in future studies on neighborhood effects of OPPs. External validation suggests the NYC latent measure successfully captured variation in drug-related public disorder. The RI measure may capture variation in the environment that is less related to policing patterns.