JMIR Publications

ABSTRACT

Background:

In occupational therapy, progress notes and other client-related administrative tasks are essential for providing treatment but are time consuming. Therapists spend at least as much time on these tasks as providing care, which contributes to growing wait lists.

Objective:

We aimed to create a custom large language model to make the process of writing progress notes more efficient by converting point-form scratch notes from pediatric occupational therapy treatment sessions into draft documentation in subjective-objective-assessment-plan (SOAP) format.

Methods:

Using a dataset of redacted historical progress notes, various training methods, including domain-adaptive pre-training and Low-rank Adaptation (LoRA) fine-tuning, were applied to train Llama 2 and 3 models. Since the historical notes lacked corresponding scratch notes, few-shot prompting with human-in-the-loop evaluations was used to generate synthetic scratch notes. This pairing of historical notes and generated scratch notes enabled effective fine-tuning of the Llama models on the desired task. The final model, a fine-tuned Llama 3 8B Instruct model, was piloted in a pediatric rehabilitation center and compared with Microsoft Copilot. Ten therapists used both models for three weeks each.

Results:

The custom model notes scored higher than manually written notes on clarity, completeness, relevance, and organization (p <.001 in each case) and similarly on conciseness. They scored higher than those from Copilot on conciseness (p <.001). However, there was no evidence from the pilot study that either model saved therapists’ time. Follow-up investigation revealed that coaching therapists to write brief scratch notes was crucial for reducing time spent on the process, but also that they tended to revert to detailed scratch notes after coaching.

Conclusions:

The model had the capacity to save time when therapists provided brief input to the model. However, in practice, therapists preferred to provide detailed input. Used in this way, the model improved note quality rather than saving time.