JMIR Publications

ABSTRACT

Background:

Large Language Models from the field of Artificial Intelligence have been one of the tools that have a great and real impact on the daily lives of people. In this regard, they emerge as an aid in specific fields such as education helping educators in cumbersome tasks such as periodical evaluations.

Objective:

This paper is focused in analyzing the benefits of Large Language Models and in particular Chain of Thought strategy for the task of evaluating the writing in Spanish of medical records by students. The aim is two-fold: first we try to save time and re-sources and second, we use the reasonings of the Chain of Thought to evaluate the ru-brics and its interpretations.

Methods:

The proposed solution studies the application of two models such as Llama 3.1 and Claude 3.5 in combination with two strategies One-Shot and Chain of Thought to evaluate how medicine students write medical records in Spanish. First, machine learn-ing metrics are applied to measure the performance of the solutions. Then, different statistical analysis is obtained at clinical record and item level. Finally, the differences between proposed models and evaluators are studied in-depth.

Results:

Machine learning metrics indicate that, at first, Claude got an Accuracy over 86% and no substantial differences among them. However, an in-depth analysis high-lights the potential of Chain of Thought both for identifying evaluator errors and for addressing issues in the rubrics. Applying this analysis, we have found 317 human errors which, after a correction, have set the Accuracy in 94.6%.

Conclusions:

Chain of Thought demonstrates strong potential for supporting the evalua-tion of clinical records written in Spanish by medical students and providing feedback to them. More importantly, it shows significant promise in assisting professors by as-sessing the quality of their rubrics and identifying possible errors.