(1553) Evaluation of Machine Learning Model Drift Across Various Races-Ethnicities in Patients With Sepsis

Introduction: Model drift, which refers to a decrease in the performance of the machine learning (ML) model over time, is well recognized. However, recent data suggests that such drift may be more pronounced in under-represented patient populations. This study aimed to evaluate the performance of a ML model over time across races, hypothesizing a decline in model performance, with a more pronounced impact on minorities.

Methods: A retrospective study that evaluated a ML model developed using the MIMIC-IV dataset, 2008-2016, to predict mortality in critically ill patients with sepsis. To examine the model drift, we evaluated the performance of the model at two time-periods, 2017-2019 and 2020-2022. From the MIMIC-IV dataset, we included adult patients who had sepsis, defined according to the SEPSIS-3 criteria. Only the first hospitalization was included. Patients with missing outcomes or demographics were excluded.Patients were aggregated into the following groups, based on their race: White, Black, Asian, and Hispanic. Model performance was measured using area under the receiver operating characteristic curve (AUROC), F1-score, and the Demographic Parity Ratio (DPR).

Results: A total of 30,371 patients met our inclusion criteria. Race distribution was 2.9% Asian, 9.2% Black, 3.5% Hispanic, and 66.1% White. The model performance was better during the 2017-2019 time-period, compared to 2020-2022, with overall AUROC of 0.847 and F1-score of 0.576 compared to AUROC of 0.795 and F1- score of 0.529. For the model performance based on race in the 2017-2019 and 2020-2022 time-periods, the AUROC was as follows: 0.851, 0.808 for Whites, 0.818, 0.788 for Blacks, 0.890, 0.792 for Asians, and 0.847, 0.846 for Hispanics.The F1-score in the 2017-2019 and 2020-2022 time-periods was 0.558, 0.499 for Whites, 0.520, 0.517 for Blacks, 0.622, 0.431 for Asians, and 0.498, 0.594 for Hispanics. The DPR for race declined from 0.317 in 2017-2019 to 0.105 in the 2020-2022 time-period.

Conclusions: In this study, we report a model drift over time which appeared to be more pronounced in subgroups of patients. Future research is needed to explore this further and identify strategies to address this in the model development and deployment.