(389) Comparative Evaluation of Machine Translation/AI Tools in International Critical Care Education

Introduction: CERTAIN (Checklist for Early Recognition and Treatment of Acute Illness and iNjury) is an established international critical care education and improvement program, which uses bilingual medical professionals to translate curriculum to different languages. The rise of machine translation (MT) including generative artificial intelligence (GenAI) offers opportunities for efficiency, yet little is known about comparative efficacy of these tools for critical care content. We present pilot methodology and results for assessing MT/GenAI in enhancing translation of critical care content across languages.

Methods: Spanish and Mandarin Chinese languages were selected for CERTAIN program needs. English curriculum were translated into Spanish and Chinese by professional translators, as well as by four MT/GenAI tools selected by literature and expert review (Google Translate, DeepL, Google Gemini, and Microsoft CoPilot). MT/GenAI translations were compared to human translation using well-established BLEU scores. For each language, all five translation versions were also evaluated in a blinded manner by three bilingual clinicians on Likert anchors of accuracy, fluency, and meaning, then made into composite scores per tool for each language. Basic tool usability was also recorded.

Results: BLEU score: Google Gemini was lowest and Google Translate highest for both languages. Spanish BLEU scores ranged from 21.57 - 51.12, and Chinese from 2.22 - 15.04.

Bilingual clinician scores: All tools performed comparably. None proved best across all domains and languages. In Spanish, Google Translate was lowest and Google Gemini highest. In Chinese, Microsoft CoPilot was lowest and human translation highest. Spanish composite scores ranged 3.95 - 4.47, and Chinese from 4.12 - 4.33.

Usability: All tools were easy-to-use. Microsoft CoPilot took noticeably longest for translations. Google Gemini added new formatting to texts.

Conclusions: This project offers a systematic approach and early data to inform use of MT/GenAI in critical care translation. Interestingly, there was discordance between BLEU and human scores. BLEU scores also were lower for Chinese than Spanish, raising methodological questions for BLEU in some languages. Future investigation is needed to refine use of MT/GenAI in enhancing critical care education.