Methods/Statistics
Dynamic Updating Strategies in Assessing Hospital Performance: A State-wide Assessment of Observed-to-Expected Mortality Ratios in Surgical Aortic Valve Replacements Jackie Pollack* Jackie Pollack Wei Yang George Arnaoutakis Michael Kallan Stephen Kimmel
Background-Prediction models that determine expected outcomes are infrequently updated, despite demonstrating increasing inaccuracy over time. This can result in incorrect assessment of hospital performance. Dynamic model updating may provide a method to ensure accurate assessments and comparisons across hospitals.
Methods-Using a baseline prediction model, derived from the Pennsylvania Health Care Cost Containment Council (1999-2006), four modeling strategies were used to estimate 30-day postoperative mortality following Surgical Aortic Valve Replacement (SAVR) from 2007 to 2018: (1) A typical, non-updating (static) model, where the model remained fixed throughout the study period. (2) Applying an annual correction factor (CF), based on the methodology of the Society of Thoracic Surgeons. (3) Calibration regression (CR) to annually recalibrate the model. (4) Dynamic logistic state space modeling (DLSSM) to continuously update model coefficients. Hospitals were ranked based on observed to expected (O/E) ratios and Z-scores to assess the impact on performance rankings and to identify outliers.
Results-The test data included 29,127 SAVRs with 765 deaths among 53 hospitals. Across all updating strategies, nearly all hospitals experienced rank changes when compared to the static model. Within ranking tertiles, 18.9% (n=10) of hospitals shifted tertile tiers with CF and CR approaches, and 22.6% (n=12) changed with DLSSM. While the static model labeled 15 hospitals as having significantly better-than-expected performance, only 3 maintained this classification with CF and DLSSM, and 5 retained it with the CR methods. No hospitals demonstrated significantly worse-than-expected mortality in the static model, but CR identified 6 and DLSSM and CF both identified 7.
Conclusion-Static models may misclassify hospital performance and rankings. Regular updating to better assess O/E ratios among SAVRs can lead to large changes in estimated performance and hospital rankings.
