JMIR Publications

ABSTRACT

Background:

OpenNotes allows patients to access their electronic health record (EHR) notes through online patient portals. However, EHR notes contain abundant medical jargon, which can be difficult for patients to comprehend. One way to improve comprehension is by reducing information overload and helping patients focus on the medical terms that matter most to them.

Objective:

In this study, we evaluated both closed-source and open-source Large Language Models (LLMs) for extracting and prioritizing medical jargon from EHR notes relevant to individual patients, leveraging prompting techniques, fine-tuning and data augmentation.

Methods:

We evaluated the performance of closed-source and open-source LLMs on a dataset of 90 expert-annotated EHR notes. We tested various combinations of settings including: i) general and structured prompts, ii) zero-shot and few-shot prompting, iii) fine-tuning and iv) data augmentation. To enhance the extraction and prioritization capabilities of open-source models in low-resource settings, we applied data augmentation using GPT-4o and integrated a ranking technique to refine the training process. Additionally, to measure the impact of dataset size, we fine-tuned the models by incrementally increasing the size of the augmented dataset from 10 to 9,995 and tested their performance. The effectiveness of the models was assessed using 10-fold cross-validation, providing a comprehensive evaluation across various settings. We report the F1 score and Mean Reciprocal Rank (MRR) for performance evaluation using two different string matching algorithms (relaxed string matching and Jaccard Index). We also conduct error analysis classifying the erroneous outputs from the models.

Results:

Our results show that open-source models achieved the highest performance, particularly when utilizing fine-tuning with gold-standard dataset. Under Jaccard Index based string matching, DeepSeek 8B set the benchmarks with an F1 of 0.431 (SD 0.046); similarly, BioMistral 7B showed an MRR of 0.577 (0.109). However, under relaxed string matching, open-source models were unable to match the performance of closed-source models, even with data augmentation or fine-tuning. We analyzed our experiment from several perspectives. First, few-shot prompting did not show an advantage over zero-shot prompting in vanilla models. Second, when comparing general and structured prompts, we found model performance could deviate largely based on prompting styles. Third, fine-tuning with a small gold truth dataset improved performance. Lastly, data augmentation yielded performance comparable to or even surpassing fine-tuning strategy. However, it also left emphasis on the importance of the quality of the augmented dataset. Conclusion: The evaluation of both closed-source and open-source LLMs highlighted the effectiveness of prompting strategies, fine-tuning, and data augmentation in enhancing model performance in low-resource scenarios.