JMIR Publications

ABSTRACT

Background:

Parkinson’s disease (PD) is a neurodegenerative disorder characterized by a wide spectrum of motor and nonmotor symptoms, making its accurate diagnosis particularly challenging. While the increasing availability of structured clinical data is broadly recognized in the field, this study specifically focused on the Parkinson’s Progression Markers Initiative (PPMI) dataset to investigate the diagnostic capacity of large language models (LLMs) for PD.

Objective:

This study aimed to investigate whether LLMs can accurately classify PD using structured clinical data, and how different prompting and fine-tuning strategies influence diagnostic outcomes.

Methods:

We analyzed structured clinical data from 1,238 participants enrolled in PPMI, using 1,052 samples for training and 186 for testing. Clinical variables were converted into natural language prompts using four formats: plain text (PT), markdown (MD), special token (ST), and markdown with special token (MD+ST). We conducted three core experiments: (1) zero- to three-shot prompting across seven LLMs from four model families, (2) supervised fine-tuning of three lightweight models, and (3) dual-output prompting using four flagship models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and LLaMA 3.3 70B) to examine whether reasoning generation influenced diagnostic classification.

Results:

Among the seven LLMs evaluated across the four prompt formats and zero- to three-shot settings, the highest F1-score (0.995) was achieved by LLaMA 3.3 70B and Gemini 1.5 Pro under few-shot prompting, both of which demonstrated perfect recall and zero inconsistency. Fine-tuned lightweight models, including GPT-4o-mini and LLaMA 3.1 8B, achieved equivalent performance (F1 = 0.995), confirming the efficiency of supervised adaptation. Under dual-output prompting, which requires models to generate both diagnostic labels and three-sentence inferences, slight declines in performance were observed (e.g., LLaMA 3.3 70B, F1 = 0.990; inconsistency = 2), suggesting that simultaneous reasoning generation may affect diagnostic reliability.

Conclusions:

Among the seven LLMs evaluated across the four prompt formats and zero- to three-shot settings, the highest F1-score (0.995) was achieved by LLaMA 3.3 70B and Gemini 1.5 Pro under few-shot prompting, both of which demonstrated perfect recall and zero inconsistency. Fine-tuned lightweight models, including GPT-4o-mini and LLaMA 3.1 8B, achieved equivalent performance (F1 = 0.995), confirming the efficiency of supervised adaptation. Under dual-output prompting, which requires models to generate both diagnostic labels and three-sentence inferences, slight declines in performance were observed (e.g., LLaMA 3.3 70B, F1 = 0.990; inconsistency = 2), suggesting that simultaneous reasoning generation may affect diagnostic reliability Clinical Trial: Among the seven LLMs evaluated across the four prompt formats and zero- to three-shot settings, the highest F1-score (0.995) was achieved by LLaMA 3.3 70B and Gemini 1.5 Pro under few-shot prompting, both of which demonstrated perfect recall and zero inconsistency. Fine-tuned lightweight models, including GPT-4o-mini and LLaMA 3.1 8B, achieved equivalent performance (F1 = 0.995), confirming the efficiency of supervised adaptation. Under dual-output prompting, which requires models to generate both diagnostic labels and three-sentence inferences, slight declines in performance were observed (e.g., LLaMA 3.3 70B, F1 = 0.990; inconsistency = 2), suggesting that simultaneous reasoning generation may affect diagnostic reliability