JMIR Publications

ABSTRACT

Background:

Identifying individuals on parole or probation can help inform care pathways and ensure continuity of care in healthcare systems. However, this sensitive justice-related information is often absent from structured electronic health record fields and instead documented in narrative clinical notes. Developing automated parole–probation detection (PPD) systems typically requires expert-annotated clinical data, but manual annotation is costly, time-consuming, and constrained by data sparsity and privacy concerns.

Objective:

This study aimed to develop and evaluate a synthetic-only training paradigm for PPD from clinical text that eliminates the need for expert-annotated training data while maintaining strong performance and cross-domain transfer to real-world data.

Methods:

We developed Synth-PPD, a synthetic dataset comprising 7,579 labeled clinical notes generated using large language models (LLMs), redundancy filtering, and automated quality evaluation. Dataset quality was assessed through human review of 1,517 synthetic notes by two annotators. The primary outcome was macro F1 score for PPD. We evaluated Synth-PPD across unsupervised and supervised learning settings using LLMs, traditional machine learning models, and transformer encoders. Performance was assessed on 293 expert-annotated real-world clinical notes from the U.S. Department of Veterans Affairs.

Results:

Synth-PPD achieved 90.05% human–LLM alignment. In the unsupervised setting, Llama 3.3 70B Instruct with chain-of-thought prompting and few-shot examples from Synth-PPD achieved a macro F1 score of 87.22%, outperforming both the corresponding zero-shot model (80.10%) and the best rule-based baseline (77.55%; all comparisons P<.001). In the supervised setting, Llama 3.1 8B fine-tuned on Synth-PPD achieved a macro F1 score of 81.43%, exceeding the best rule-based baseline (77.55%; P<.001) and achieving a numerically higher score than the substantially larger zero-shot Llama 3.3 70B Instruct model (80.10%; P=0.08). In contrast, traditional machine learning models and transformer encoders exhibited greater performance degradation under domain shift and derived limited benefit from increasing synthetic training data. Additional experiments showed that increasing synthetic training data improved cross-domain performance for LLM-based models but yielded comparatively smaller gains for traditional approaches.

Conclusions:

Synthetic data generated by LLMs can serve as a scalable, cost-effective, and privacy-preserving alternative to expert-annotated real-world clinical data for parole–probation detection from clinical text. The findings demonstrate successful cross-domain transfer to real-world VA clinical notes.