Big Data/Machine Learning/AI
Auditing Models: Machine Learning for Early Detection of Prediction Errors Kevin Anderson Ruperto Mateo Panduro* Kevin Anderson Ruperto Mateo Panduro Roberta Moreira Wichmann Alexandre Chiavegatto Filho
The ability to anticipate prediction errors in clinical models is essential for improving decision-making and optimizing resource allocation, especially in high-pressure scenarios such as the COVID-19 pandemic. This study aimed to develop and evaluate machine learning algorithms to predict errors in a base model designed to classify critical outcomes, specifically mortality and ICU admission, in patients diagnosed with COVID-19. Data from 8,477 hospitalized patients were obtained through the Artificial Intelligence for COVID-19 in Brazil (IACOV-BR) network, covering 18 hospitals across the country. A base predictive model was built using XGBoost, achieving an AUC of 0.852 for mortality and 0.928 for ICU admission. Auxiliary models were subsequently developed to predict type 1 (false positives) and type 2 (false negatives) errors. The XGBoost algorithm demonstrated superior performance, with AUCs ranging from 0.662 for type 1 error prediction in mortality to 0.870 for type 2 error prediction in ICU admission. Group analysis identified intersections where the base model’s predictions exhibited higher-than-average error rates, particularly among groups with high predictive probabilities in the auxiliary models. These findings underscore the importance of integrating auxiliary models to detect high-risk prediction errors, contributing to more reliable clinical decision-making and improved patient management during healthcare emergencies.