(384) Evaluating Data Fidelity: An Innovative Tool for Building High-Quality AI Models in Critical Care

Introduction: High data fidelity is key to building robust and reliable artificial intelligence (AI) healthcare tools. Poor fidelity impairs the generalizability of AI models and propagates harmful biases during implementation. Despite availability of large electronic health record (EHR) datasets, there is a paucity of robust tools to quantify and compare data fidelity among databases and organizations. Current tools have limited interpretability and inconsistent inclusion of quality dimensions. We created a novel tool to score data fidelity based on all reported dimensions of fidelity allowing for comparison between institutions or datasets.

Methods: We used the CURE ID database to evaluate seven fidelity dimensions based on expanded Weiskopf classification: completeness, correctness, concordance, recency, plausibility, conformance, and bias. Each dimension was quantified using binary or continuous scoring, providing institution-level dimensional fidelity scores. We built a binary logistic regression model using mortality as a ground truth value and calculated a fidelity score for each institution based on the model’s coefficients for each dimension.

Results: CURE ID includes 55,899 patients from 8 institutions. We found notable deficits in completeness for gender, race, ethnicity related elements; correctness and plausibility in age and date of birth; and discordance between data categories reporting mortality. Demographics distribution was skewed compared to census related benchmarks. The model identified external concordance and completeness as the key dimensions influencing data quality, with importance scores of 1.06 and 0.23, respectively. The aggregate fidelity scores computed for each institution ranged from 1.14 to 2.18 where lower score indicates better data fidelity.

Conclusions: Our approach to quantifying data fidelity provides a framework for multidimensional self-assessments and comparisons among databases and institutions. Continued research in this area should include more heterogeneous datasets, nuanced outcome selection, and more robust measurement techniques in each quality dimension. The tool is limited by the need for preprocessed data or a Common Data Model structure, but its maturation can fundamentally improve multi-center research reliability and the quality of related AI models.