(853) Assessment of Diagnostic Calibration in PICU Providers: A Clinical Vignette Pilot Study

Introduction: Diagnostic errors are a notable cause of preventable patient harm. Within pediatrics, they were identified in 2019 as a priority research topic by the Children’s Patient Safety Network. There is a paucity of information on cognitive factors that contribute to diagnostic errors by physicians and advanced practice practitioners (APPs) in the PICU. This aim of this study was to define diagnostic calibration habits (relationship between accuracy and confidence) in pediatric intensivists, fellows, and APPs.

Methods: Pediatric intensivists, fellows, and APPs at the University of Alabama at Birmingham completed four clinical case vignettes through an anonymous web-based survey using REDCap (Vanderbilt University). The cases had four phases each (history, physical exam, general labs and imaging, and definitive labs and imaging) to simulate the progression of a patient presentation and diagnostic development. Participants were evaluated on diagnostic accuracy, confidence level in the reported diagnosis, diagnostic calibration, and miscalibration over the course of each case.

Results: A total of 13 pediatric intensivists, 7 pediatric critical care fellows, and 3 certified registered nurse practitioners showed both improvement in accuracy (14% vs. 82%) (p < 0.001) and increased confidence (4.8 vs. 8.6 out of 10) (p < 0.001) with progression of the cases. There was a negligible difference in diagnostic calibration between the phases (0.28 vs. 0.14) (p=0.11). However, miscalibration, characterized more specifically as overconfidence, improved as cases progressed (0.32 vs 0.03) (p=0.03).

Conclusions: Our study suggests that PICU prescribers maintain reasonable diagnostic calibration throughout the course of a patient case, with notable improvement in accuracy as confidence increases. Participants erred most typically with overconfidence in their diagnosis, although this improved with case progression as more information was revealed. This needs to be further studied in a larger, more heterogenous population of PICU prescribers.