(1461) Are Discharge Notes Enough?

Introduction: The recent surge in AI has sparked significant interest in automating administrative tasks in medicine. One crucial area of exploration is the direct deduction of billing codes from clinical documents like discharge summaries, thereby alleviating clinicians of this clerical burden. This transition could reduce physicians' burnout rates and allow them to dedicate more time to patient care. Our research aims to determine the effectiveness of the most advanced Large Language Models (LLM) in predicting accurate billing codes from discharge notes. Additionally, we seek to uncover any disparities between patient ethnicities and genders. Our analyses are centered on patients diagnosed with Sepsis, which is typically well-documented and can be confirmed from blood culture results and antibiotic use. The findings of this research could significantly influence the future of AI applications in medicine.

Methods: We drew our data from the Medical Information Mart for Intensive Care III (MIMIC) dataset, one of the US's largest and best-maintained clinical data repositories. We used the following three methods to filter for patients with sepsis: Diagnosis codes: Patients billed with ICD9 codes of 995.91 and 995.92. Discharge summaries were analyzed by two LLMs: Amazon’s Comprehend Medical and Open AI’s ChatGPT. Treatment Data: Patients were identified as having sepsis when their labs showed a positive blood test. The prescription data showed that their antibiotic regimen ended more than 48 hours after admission.

Results: 5,184 patients were identified with a diagnosis of sepsis using ICD codes. Out of these patients, LLM was initially run on 180 patients due to processing and cost restraints. The LLMs performed worse than we had hoped, ranging from 61% sensitivity with Amazon’s model to only 13% with Open AI’s model. There was no significant difference in the rate at which the LLMs correctly identified sepsis between ethnicities and genders.

Conclusions: Discharge summaries are not currently adequate for providing billing codes. Large Retrieval-Augmented Generation (RAG) may improve performance along with different types of clinical documentation. Future work is needed to improve model build and the sensitivity of natural language processing to extract and bill for various diagnoses better.