JMIR Publications

ABSTRACT

Background:

Accurate reporting in nuclear medicine is essential for clinical decision- making. Trainees often generate preliminary reports with variable quality, and AI tools such as ChatGPT may enhance report clarity and accuracy particularly in the impression section of the report.

Objective:

To evaluate and compare the quality of PET/CT report impression sections generated by trainees and by the AI chatbot ChatGPT, focusing on correctness, clarity, completeness, organization, use of diagnostic certainty terminology, and physician satisfaction.

Methods:

We compared the impression section of 200 PET-CT reports, which was generated by trainees and AI (n=100) each. The AI-generated the impression based on the stem, clinical history and technical section of the trainees generated reports. Reports were blindly rated by three nuclear medicine physicians. A survey of questions including Likert-scale questions assessed correctness, certainty word use, clarity, completeness, organization, and satisfaction. Statistical analyses included Fisher’s test, t-tests, chi- square, ANOVA, and effect sizes (Cohen’s d). Thematic analysis was performed on free- text comments.

Results:

- Correctness: Comparable between groups (AI: 92% correct, mean=0.92 ±0.27; trainees: 91% correct, mean=0.91 ±0.29; p=0.80, negligible effect). - Certainty word use: AI included certainty terms in 94% of reports versus 81% for trainees (p=0.005, χ²=6.58, small effect size). - Type of certainty word: AI used higher-level certainty terms more frequently (mean=4.23 vs 3.69 on 5-point scale; p=0.02). - Clarity: High in both groups (AI 4.49 ±0.70, 89.8% rated clear vs trainees 4.35 ±0.85, 87%; p=0.20). - Completeness: AI significantly outperformed trainees (AI 4.52 ±0.76, 90.4% complete vs trainees 4.09 ±1.01, 81.8%; p<0.001, small–medium effect). - Organization: Similar (AI 4.22 vs 4.39; p=0.15, not significant). - Satisfaction: Similar (AI 4.31 ±0.77, 86.2% satisfied vs trainees 4.14 ±0.91, 82.8%; p=0.16). Thematic analysis showed trainees were more often criticized for Accuracy (74% vs 19%, p<0.001) and Actionability (20% vs 4.8%, p=0.02), while AI was flagged for long impression (33% vs 0%, p<0.001).

Conclusions:

These findings suggest that AI chatbot, Chat GPT can serve as a valuable adjunct in nuclear medicine reporting, particularly at the resident level, by improving decisiveness and completeness while complementing trainee education and clinical oversight. Clinical Trial: Not applicable. This study did not involve a prospective clinical trial.