(1086) LLM Performance on Board Review in Critical Care (LLM-BRiCC)

Introduction: Large language models (LLMs) have demonstrated strong performance on general knowledge medical exams (e.g., USMLE Step 1 and 2), as well as board exams in some clinical specialties. However, LLM performance on critical care board questions has not been evaluated; this performance could be affected by the complexity of clinical decision-making and frequent need for interpreting multimodal data in critical care. We evaluated the performance of LLMs on both text-based and multi-modal questions and compared LLM performance to the pooled performance of other examinees.

Methods: In total, 200 board review questions from the Society of Critical Care Medicine (SCCM) question bank were used, 172 with text only, and 28 multi-modal questions. Using a one-shot approach, each question was individually inputted into ChatGPT-4 (GPT-4) preceded by a standardized prompt: "Please answer this multiple-choice critical care board exam question and provide an explanation to support your answer to educate critical care providers on the rationale for your answer." The LLM’s answer was compared to the correct answer, and paired t-tests were used to compare GPT-4 performance to mean clinician performance. Performance was also compared between GPT-4 and clinicians within question categories.

Results: GPT-4 correctly answered 86.5% of questions, compared to clinician performance of 74.8% (p < 0.001), with similar directionality of performance in the subset of 28 multi-modal questions (85.7% vs 76.6%, p=0.2). GPT-4 performance was better than clinicians in all question categories, with the biggest discrepancy in Endocrine (100 vs 73%, 27% higher performance, N=9 questions), Pharmacology/Toxicology (100 vs 80%, 20% higher performance, N=10), and Infectious Disease (92 vs 73%, 19% higher performance, N=25).

Conclusions: We found that GPT-4 outperformed average clinician performance in both unimodal (text only) and a small subset of multi-modal questions, and across all categories. The exact role of LLMs as a study tool with the changing nature of knowledge assessment remains to be evaluated in future work.