(1122) Automated Abstraction of Pulmonary Embolism Concepts From CT Reports With Large Language Models

Introduction: Pulmonary embolism (PE) is a substantial cause of mortality in critical care. It is diagnosed with a computed tomography (CT) scan with PE protocol-timed contrast (CT PE). Machine learning methods have been used to identify PE concepts from reports, but suffer from limitations in more complicated abstractions. It is unknown how large language models (LLMs) perform in extracting data for PEs.

Methods: CT PE reports were extracted from MIMIC-IV and Duke Health’s electronic health record (EHR) database. 21 concepts were agreed upon by study members. A concept could be abstracted as present, absent, or unknown. Two raters rated in blocks of 50 reports each, up to 2 blocks. Cohen’s kappa (k) was calculated for each interrater pair. High agreement was defined as k > 0.8 and moderate agreement as k between 0.6 and 0.8. PE reports were then supplied to Llama3.1:70b using a custom prompt with Ollama. The performance of Llama was evaluated by comparing its extractions to the elements consistently agreed upon by the two human raters.

Results: 9,132 CT PE reports were extracted from MIMIC-IV across 4,994 patients (60.5% Female) and 5,443 hospitalizations. 82,524 CT PE reports were extracted from Duke Health across 61,027 patients (40.9% female) and 81,106 hospitalizations. From MIMIC-IV, 246 reports were reviewed by two raters with high agreement on five concepts: PE positive, negative, proximal, lobar, or segmental. Compared to Llama3.1, two concepts had moderate agreement. Assuming missing data as negative, five concepts had high agreement and five moderate.

From Duke Health, 184 reports reviewed by two raters showed moderate agreement on three concepts. Compared to Llama3.1, one concept had high agreement and one moderate. Assuming missing data as negative, three concepts had high agreement and seven moderate.

Conclusions: Llama3.1 effectively abstracts concepts from CT PE reports, achieving high agreement with human raters for several concepts. Limitations include defining a gold standard and addressing concepts with poor agreement. Despite these challenges, LLMs show promise for building datasets and automating EHR workflows. Future efforts should focus on expanding concept evaluation and fine-tuning the model to enhance accuracy and reliability.