(353) Large Language Models for Mortality Prediction Using Structured EHR and Unstructured Clinical Notes

Introduction: Artificial intelligence (AI) has been used to predict mortality in intensive care unit (ICU) patients using structured/ unstructured data. However, the use of large language models (LLMs) has not been studied extensively. The aim of our project was to demonstrate the potential of using LLMs for mortality prediction in the ICU.

Methods: All adult patients in the Medical Information Mart for Intensive Care (MIMIC)-III dataset were included in the study. Patients with multiple ICU admissions during the same hospital stay, ICU stay less than 24 hours, patients with missing data in the first 24 hours or those expired within first 48 hours of ICU admission were excluded. Structured EHR data included patient's age, gender, race, comorbidities, and ICU data from the first 24 hours ( vital signs, laboratory results, severity scores). Unstructured clinical notes included physician, nursing notes and radiology reports from the first 24 hours of admission. The target variable was short-term mortality (3 days post first 24 hours of admission) and long-term mortality (3-30 days post first 24 hours of admission). For structured EHR data, two deep learning models were used: neural network (NN) and transformer. For clinical notes, five NLP and LLM models were used: bag-of-words, ClinicalBERT, GatorTron, Meditron, and OpenAI. Models were trained using supervised learning with three different settings: structured EHR, clinical notes, and structured EHR/clinical notes combined.

Results: A total of 29,353 patients were included in the final cohort. The dataset was divided into train, validation and test sets (80:10:10). The results for short-term mortality prediction showed that combination of structured EHR/clinical notes using OpenAI/Transformer yielded the highest area under the receiver operating characteristic curve (AUROC) with 0.90 (95% confidence interval 0.87-0.93). Similarly, combination of structured EHR/clinical notes using Meditron/Transformer and GatorTron/Transformer both yielded the highest AUROC for long-term mortality prediction with 0.85 (95% confidence interval 0.83-0.87).

Conclusions: There is a potential for integrating LLMs with structured EHR data and unstructured clinical notes to improve mortality prediction in the ICU. This approach can be applied to other ICU outcomes predictions.