JMIR Publications

ABSTRACT

Background:

The growing administrative burden on clinicians, particularly in medical documentation, contributes to burnout and may compromise patient safety. Recent advancements in generative artificial intelligence offer a promising solution to improve documentation processes and adress these challenges.

Objective:

This study aims to evaluate the feasibility of using a fine-tuned OpenAI Curie model to automate the generation of medical report summaries (epicrises) in ophthalmology. By assessing the model’s performance through human and automated evaluations, the research seeks to determine its potential for reducing clinician workload while ensuring accuracy, usefulness, and compliance with regulatory requirements.

Methods:

A dataset of around 60,000 anonymized medical letters was created using a custom algorithm to comply with GDPR. The Curie model was fine-tuned on this dataset to generate epicrises from medical histories, diagnoses, and findings. Performance evaluation involved various human assessments and automated evaluations from two LLMs.

Results:

The fine-tuned Curie model effectively generated coherent medical summaries. Human evaluators (n=769) rated the majority of the AI-generated epicrises as helpful or excellent. Assessments indicated varying agreement on aspects like correctness, accuracy, usefulness, and potential time savings, estimated to be up to 50 seconds per report. Automated evaluations generally aligned with human ratings.

Conclusions:

This research emphasizes AI’s potential to streamline documentation workflows in ophthalmology. However, significant legal and regulatory challenges exist, particularly regarding data transfer outside the EU. The model performed better on common cases, underscoring the importance of human review to maintain quality and safety.This study highlights the promise of transformer-based LLMs in reducing administrative tasks in healthcare. It outlines a pipeline for integrating LLMs into EU clinical practice, emphasizing the need for careful implementation to ensure efficiency and patient safety.