(1536) Development and Validation of a Pediatric ICU Deterioration Prediction Model Using Machine Learning

Introduction: Situation awareness and just-in-time training in the pediatric ICU (PICU) require accurate and timely identification of children at risk for deterioration. Existing systems are limited by low sensitivity or low positive predictive value. We hypothesized that a multivariable model, using EHR data, could predict deterioration in the PICU with a sensitivity >0.8 and positive predictive value (PPV) >0.25.

Methods: We developed and validated a PICU Warning INdex (PWIN) deterioration prediction model at a single children’s hospital. We included PICU admissions for patients < 24 years from 2014-2023. We excluded admissions with length of stay < 24 hrs or had deterioration < 8 hrs from admission. The primary outcome was deterioration (unplanned intubation, cardiac arrest, ECMO cannulation, and/or dilute epinephrine (10mcg/ml) bolus). We excluded events < 8 hrs from a prior event. We trained an ensemble model of six LASSO regression classifiers predicting deterioration at 1,2,4,6,8, 12-hr horizons. We performed nested 10-fold cross validation and reported average performance across 10 test folds. Regularization strength hyperparameter was optimized using grid search cross validation. Metrics were sensitivity, PPV, area under the ROC curve (AUROC), area under the precision-recall curve (AUPRC), and scaled Brier score (SBS) at 1,4, and 12-hr horizons (reported as estimate[95%CI]).

Results: Of 35,964 PICU admissions, 28,272 met inclusion criteria, and 1,109 (3.9%) had deterioration. PWIN used 6,148 features from EHR data (vital sign, laboratory, diagnosis, and medication data) and achieved hypothesized sensitivity (1h: 0.87 [0.82-0.92]; 4h: 0.82 [0.77-0.88]; 12h: 0.77 [0.71-0.84]) and PPV (1h: 0.25 [0.23-0.27]; 4h: 0.25 [0.23-0.27]; 12h: 0.25 [0.23-0.27])) at all horizons. The model had excellent AUROCs (1h: 0.93 [0.91-0.95]; 4h: 0.92 [0.89-0.94]; 12h: 0.90 [0.87-0.92]), adequate AUPRCs ((1h: 0.61 [0.55-0.69]; 4h: 0.56 [0.49-0.63]; 12h: 0.51 [0.44-0.59]), and fair calibration (SBS: 1h: 0.42 [0.36-0.48]; 4h: 0.38 [0.32-0.44];12h: 0.34 [0.28-0.40]).

Conclusions: PWIN predicted deterioration with high sensitivity and PPV to enhance situation awareness and operationalize just-in-time training. Next steps include prospective validation, multicenter assessment, and comparison to bedside clinician assessments.