Big Data/Machine Learning/AI
What if we had built a prediction model with a survival super learner instead of a Cox model 10 years ago? Arthur Chatton* Arthur Chatton Héloïse Cardinal Kevin Assob Feugo Robert W Platt Mireille E Schnitzer
Classical approaches to prediction were mainly based on parametric models, but there is a current trend towards using more flexible machine learning approaches. Indeed, many machine learning approaches do not rely on parametric assumptions about the underlying data-generating process and have the potential to predict more accurately. Nevertheless, there is still no free-lunch method, even with machine learning. Their improved flexibility is at the cost of an increased risk of over-fitting, needing more data to converge, and the final performance still depends on the particular data-generating process. Ensemble methods aim to overcome these issues by combining a diverse and rich set of algorithms or “learners” into one final prediction. For instance, the super learner takes a weighted sum of the individual predictions from each incorporated learner to produce a final prediction that is theoretically as accurate as the best-performing candidate learner.
However, both the patients’ characteristics and the guidelines to treat them evolve over time. As a result, the performance of a prediction model decreases over time with this population shift, and a prediction model performing well when developed (even externally validated) can become useless later.
Little evidence exists about the performance drift occurring over time for machine learning-based prediction models. But ensemble methods were not considered, nor were time-to-event outcomes. Therefore, the present study aims to fill this gap in the literature.
The KTFS is a prediction score developed in the early 2000s in France to predict the return to dialysis eight years after kidney transplantation. We develop a KTFS-like score with a survival super learner on the same learning data as the KTFS used, and we validate their performance (discrimination, calibration and net benefit at 8 years) on patients transplanted between 2010 and 2015 from the same open cohort and other European or Canadian external cohorts.