(1349) Challenges With Updating APACHE During the COVID-19 Pandemic

Introduction: Medical care evolves over time, and thus causes calibration drift in benchmarking tools such as APACHE. While APACHE IVb, released in 2016, was due for updating, the pandemic distorted relationships between risk factors and outcomes. Standardized mortality ratios (SMRs) temporarily increased in both COVID and non-COVID patients calling into question the suitability of creating new models from 2020-22 data. A method was needed to censor unreliable data.

Methods: The APACHE Outcomes database has detailed data on 864,881 patients from 112 hospitals between 2016 to 2023. Calibration curves were generated using monthly data, plotting Day 1 Acute Physiology Score (APS) against ICU mortality. Discrimination and calibration were determined for the pre-, intra-, and post-pandemic periods.

Results: An APS of 80 normally predicts 30% ICU mortality but during the pandemic observed mortality was almost 50% higher. Plotted lines overlapped pre- and post-pandemic, but were outliers for 25 months beginning in March 2020. Relationships normalized in April 2022 co-incident with Omicron emerging as the dominant strain of COVID. With censoring, the patient count was reduced by 249,323 to 615,588. In the pre-, intra- and post-pandemic periods discrimination (AUROC) was 0.91, 0.89, 0.90, calibration (Brier Score) was 0.05, 0.07, 0.05, and Standardized mortality ratios (SMR) were 1.01, 1.14, 0.99, respectively.

Conclusions: Benchmarking models benefit from the largest possible sample, but COVID-19 introduced distortions for 25 months. The APACHE 5 models will thus be developed and validated on a split sample that excludes data from the height of the pandemic.