JMIR Publications

ABSTRACT

Chronic dermatologic diseases such as pemphigus require long-term follow-up, generating extensive longitudinal clinical documentation that is difficult to review comprehensively during routine visits and increasing clinician workload as well as the risk of missing critical historical information. We evaluated whether a locally deployed, privacy-preserving small language model (SLM) could retrieve structured clinical features and generate longitudinal summaries from long-term dermatology follow-up records. In this retrospective case series, 30 patients with pemphigus contributed 541 visit notes that were aggregated into full longitudinal records (89,336 words); 55 clinically relevant features were annotated by two expert dermatologists. The locally deployed SLM (Qwen3 4B Thinking 2507) was queried with each complete record to retrieve 55 features and generate one final report summary. Across feature retrieval tasks, mean accuracy was 82.25±0.17%. Dermatologists’ ratings of AI-generated summaries were high for overall quality (8.23 for Evaluator 1 and 8.47 for Evaluator 2), clinical accuracy (7.93 and 8.20, respectively), and usefulness (8.50 and 8.47, respectively), with no statistically significant differences in mean ratings between evaluators. In a separate, independently rated pairwise comparison, AI-generated reports were preferred in 53.3% of evaluations. These findings suggest that privacy-preserving, locally deployed SLMs can generate clinician-facing longitudinal summaries that receive high expert ratings for overall quality, clinical accuracy, and usefulness when reviewed in the context of the patients’ original longitudinal follow-up documentation