(1308) Do Critical Care Predictive Models Exhibit Consistent Performance Across Patient Risk Tiers?

Introduction: While risk-adjusted models are commonly developed to predict critical care outcomes, there has been limited research on evaluating their performance within independently defined risk tiers (by clinicians/other models), particularly for high and low-risk patients. This study seeks to evaluate the performance of a set of ICU outcome prediction models against risk tiers defined by the APACHE IVa score.

Methods: The dataset, comprised of 184,895 ICU stays in a US-based healthcare system acquired through the Philips eICU program between 2005 and 2024, was divided into risk tiers calculated according to APACHE IVa mortality score deciles. Data from the first 24 hours for each patient was used to calculate predicted outcomes for ICU mortality, ICU/hospital length of stay, and ventilation probability and duration using machine learning models, then compared to actual values. Performance was evaluated using A/P (Actual/predicted) ratio and AUROC (area under the receiving operator curve) for mortality and ventilation predictions, along with MAE (mean absolute error) for ICU and hospital length of stay and ventilation duration for each risk decile.

Results: A/P ratios generally displayed a rising trend from low to high-risk deciles for all models except ICU LOS. However, ranges varied across the models: [0.34, 0.90] for mortality, [0.54, 1.20] for ventilation probability, [0.36, 1.52] for ventilation duration, [0.93, 1.1] for ICU LOS, and [1.13, 1.29] for hospital LOS. As risk increased through the deciles, the models showed decreasing AUROC overall, remaining in the range of [0.76, 0.85] for the mortality and [0.83, 0.95] for the ventilation probability model. Conversely, as risk increases, MAE rose from 10 to 92 hours for the ventilation duration model, from 24 to 104 hours for ICU LOS model and from 60 to 141 hours for the hospital LOS model.

Conclusions: While critical care models are typically evaluated based on risk tiers determined by the specific outcomes, they are seldom compared against other risk tiers established by clinical and other models. This study demonstrated the potential variability in performance and highlighted the importance of assessing model performance across patient risk levels.