JMIR Publications

ABSTRACT

Background:

Language barriers contribute significantly to healthcare disparities in the United States, where a sizeable proportion of patients are exclusively Spanish-speaking. In orthopedic surgery, such barriers impact both patient comprehension and patient engagement with available resources. Previous studies have explored the utility of large language models (LLMs) for medical translation but have yet to robustly evaluate AI-driven translation and simplification of orthopedic materials for Spanish speakers.

Objective:

This study utilizes the Bilingual Evaluation Understudy (BLEU) method to assess translation quality and investigates the ability of AI to simplify patient education materials (PEMs) in Spanish.

Methods:

PEMs (n = 78) from the American Academy of Orthopaedic Surgery (AAOS) were translated from English to Spanish using two LLMs (GPT-4 and Google Translate). The BLEU methodology was applied to compare AI translations with professional human-translated PEMs. Friedman’s test and Dunn’s multiple comparisons test were used to statistically quantify differences in translation quality. Readability analysis and feature analysis were subsequently performed to evaluate text simplification success and the impact of English text features on BLEU scores. The capability of an LLM to simplify medical language written in Spanish was also assessed.

Results:

As measured by BLEU scores, GPT-4 showed moderate success in translating PEMs into Spanish but was less successful than Google Translate. Simplified PEMs demonstrated improved readability compared to original versions (P<.001) but were unable to reach the targeted grade-level simplification. Feature analysis revealed that total syllables and average syllables per sentence had the highest impact on BLEU scores. GPT-4 was able to significantly reduce the complexity of medical text written in Spanish (P <.001).

Conclusions:

While Google Translate outperformed GPT-4 in translation accuracy, LLMs such as GPT-4 may provide significant utility in translating and simplifying medical texts into Spanish. We recommend considering a dual approach of using Google Translate for translation and GPT-4 for simplification in order to improve Spanish-speaking patient accessibility and education in orthopedic surgery.