JMIR Publications

ABSTRACT

Background:

Clinical Temporal Relation Extraction (CTRE) is essential for reconstructing patient timelines from unstructured Electronic Health Records (EHRs). However, the linguistic complexity of clinical notes and the high cost of expert annotation impede the development of large-scale training corpora. While Large Language Models (LLMs) have transformed general Natural Language Processing, their application to CTRE remains underexplored.

Objective:

This study aims to determine the optimal adaptation strategy for CTRE by conducting a comprehensive benchmarking of LLM architectures and fine-tuning methodologies in both data-rich and limited-data regimes.

Methods:

We evaluated four LLMs representing two distinct architectures: Transformer Encoders (GatorTron-Base, GatorTron-Large) and Transformer Decoders (LLaMA 3.1-8B, MeLLaMA-13B). We compared four adaptation strategies: (1) Standard Fine-Tuning, (2) Hard-Prompting, (3) Soft-Prompting, and (4) Low-Rank Adaptation (LoRA). Experiments were conducted on the 2012 i2b2 CTRE benchmark in both full-supervision and 1-shot scenarios.

Results:

We achieved results that exceed the current state-of-the-art (SOTA) on the 2012 i2b2 dataset. Comparative analysis reveals that hard-prompting consistently yields superior efficacy compared to standard fine-tuning. Regarding Parameter-Efficient Fine-Tuning (PEFT) strategies, Low-Rank Adaptation (LoRA) targeting query and value layers emerged as the optimal configuration. Conversely, soft-prompting demonstrated suboptimal performance, likely due to constraints on representational capacity. Architecturally, we observed a performance dichotomy based on data availability: Encoder-based models (GatorTron) exhibited superior stability and accuracy in few-shot scenarios, whereas Decoder-based models (LLaMA 3.1, MeLLaMA) demonstrated dominant performance in data-rich regimes.

Conclusions:

This study provides a rigorous roadmap for adapting LLMs to clinical extraction tasks. Based on our empirical findings, we recommend hard-prompting to maximize predictive accuracy and identify specific LoRA configurations (targeting query and value layers) as the preferred approach when computational efficiency is paramount. Furthermore, our findings suggest that while generative Decoders excel with abundant data, domain-specific Encoders remain the robust choice for few-shot clinical applications.