JMIR Publications

ABSTRACT

Background:

One of the barriers to the wider adoption of clinical information modelling standards is the high cost associated with the time and effort required to model and map data from legacy systems, as well as the need for specialized modelling experts. Meanwhile, Large Language Models (LLMs) are becoming powerful tools for data transformation and semantic processing.

Objective:

The objective of this work was to evaluate the capability of LLMs to support the mapping of structured clinical data to HL7 FHIR resources and attributes.

Methods:

Using data from the Medical Information Mart for Intensive Care IV, a data pipeline integrating clustering techniques, retrieval-augmented resource grounding, and schema-guided prompting was implemented to support the FHIR mapping of structured clinical data. The pipeline was evaluated under two scenarios with different levels of complexity. In the first scenario, used as a baseline, the input data was well described and contextually rich. In the second scenario, simulating a more realistic situation, the input data lacked both descriptions and contextual information.

Results:

Using 65 data elements, very high resource-identification accuracy was obtained in the baseline scenario. For GPT-4o, attribute-level accuracy reached 80.61%, whereas for Llama 3.3 70B it reached 69.53%. In the Real-World scenario, GPT-4o performance decreased to 94% for resource identification and 75.23% for attribute-level selection, reflecting the impact of reduced contextual information, although refined prompts preserved stable performance across temperature settings. For Llama 3.3, the corresponding attribute-level result was 61.84%. Error analysis showed that most deviations arose from terminal-level semantic ambiguity and cross-resource confusion, whereas hallucinated attributes and structural mismatches were infrequent and were mitigated by schema-informed prompting.

Conclusions:

This work empirically showed that LLMs could assist in the mapping of structured clinical data to HL7 FHIR resources and attributes, but the main remaining challenge is fine-grained semantic disambiguation. Across both evaluation scenarios, metadata availability, retrieval quality, and prompt design influenced performance primarily by shaping the contextual grounding available to the model before the final mapping decision. In context-rich settings, schema-constrained prompting improved attribute-level mapping, while residual errors concentrated mainly on ambiguity at the terminal-element level rather than on structural or formatting failures. In sparse and heterogeneous settings, clustering-based context recovery substantially improved Top-k resource identification and yielded corresponding gains in downstream attribute mapping, although errors continued to cluster around confusion between semantically adjacent resources and terminal-field ambiguity.