Currently submitted to: JMIR Formative Research
Date Submitted: May 29, 2026
Open Peer Review Period: Jun 9, 2026 - Aug 4, 2026
(currently open for review)
Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
Large Language Models for Automated Structured Information Extraction From CT Reports in Patients With Lymphoma: Retrospective Single-Center Study
ABSTRACT
Background:
Radiology reports are commonly written as free text, which limits data standardization, automated analysis, and secondary use for research and clinical decision support. Structured reporting templates may improve consistency and interoperability, but their adoption can be limited by increased reporting burden and reduced narrative flexibility. Large language models may support the automatic extraction of structured information from existing free-text radiology reports.
Objective:
To compare different large language models (LLMs) for natural language processing of free-text CT reports in patients with lymphoma, assessing their ability to automatically populate a standardized structured reporting (SR) template.
Methods:
In this single-center retrospective study, 174 Italian CT reports for lymphoma staging were analyzed. The Report section of the SIRM lymphoma CT SR template (65 fields: 19 free-text, 29 numerical, 17 multiple-choice) was used as reference structure. Two radiologists generated reference annotations and inter-annotator agreement was assessed with Krippendorff’s alpha. Claude 3.5 Sonnet, Gemini 1.5 Flash, and Mistral Large 2407 were tested in two settings: full schema and lesion-filtered schema. Performance was evaluated using accuracy, specificity, sensitivity, and F1-score.
Results:
Mean global inter-annotator agreement was 0.75±0.13. All models performed better with lesionfiltered schemas than with full schemas: mean accuracy increased from 0.63–0.64 to 0.72–0.74 and F1-score from 0.52–0.54 to 0.76–0.78, with specificity remaining >0.82. Multiple-choice fields showed the best performance (filtered-schema F1 >0.90; sensitivity ≥0.95), followed by numerical fields (F1 ~0.80), while free-text fields were the most challenging (F1 ~0.57). Claude 3.5 Sonnet achieved the best overall results (accuracy 0.74; F1-score 0.78), closely followed by Gemini 1.5 Flash.
Conclusions:
General-purpose LLMs can effectively extract structured information from Italian lymphoma CT free-text reports without domain-specific fine-tuning. Schema filtering substantially improves performance and may facilitate integration of LLM-based tools into radiology workflows for SR completion, research data reuse, and clinical decision support.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.