(388) Using Large Language Models to Annotate Substance Use Behavior From ICU Patients' Clinical Notes

Introduction: Critically ill patient’s electronic health record can have high volumes of complex data. Accurate annotation of this information is crucial for patient monitoring, treatment planning, and public health studies. Traditional methods for annotating clinical notes typically involve manual review, which is time-consuming and prone to human error. Socia; determinants of health such as substance use is a major comorbidity for critically ill patients.It can be challenging to automatically extract from clinical notes,especially based on the timing of patient’s use. In this study, we explore the capabilities of large language models (LLMs) in annotating patient substance use behavior from the clinical notes of ICU patients.

Methods: We utilized the publicly available MIMIC-SBDH (Social and Behavioral Determinants of Health), which is derived from MIMIC-III discharge summaries of 7025 patients treated in ICU. Substance use behavior was manually annotated into five categories as ground truth for our study: Present, Past, Never, Unsure, and nan(no data). To automatically extract information, we used the 8-billion-parameter model from the Llama 3 model. For model evaluation, we calculated the match score, which measures the alignment between the original labels and the generated values, quantifying the proportion of cases where the generated labels matched the original ones.

Results: We found that the model achieved high match scores for the Past category annotations, with 83.26% for drug behavior, 88.74% for alcohol behavior, and 90.62% for tobacco behavior. However, the model performed poorly for the Unsure category , scoring 2.71% for alcohol, 2.25% for tobacco, and 3.47% for drugs. When combining all categories (Present, Past, Never, Unsure, and nan), the model's overall accuracy was 51.70% for alcohol, 31.37% for drugs, and 72.62% for tobacco.

Conclusions: Large language models seem to reliably capture components of a patient's social determinants of health,except when it is not clearly documented in clinician notes(Unsure categor). Our experimentation provides early insights into the opportunities to use LLMs for automated information extraction from complex healthcare data such as that of critically ill patients.