JMIR Publications

ABSTRACT

Background:

Oral cancer treatment planning must reconcile domestic guidelines with emerging evidence and regulatory updates while protecting patient privacy. Cloud-hosted large language models (LLMs) are difficult to deploy in hospitals because of data governance, cost, and misalignment with country-specific rules.

Objective:

This study aimed to develop an on-premises, issue-driven multi-agent LLM system (OncoRAG-Oral) that combines local retrieval-augmented generation (RAG) over Japanese guidelines with de-identified PubMed searches, and to evaluate its feasibility retrospectively.

Methods:

Development proceeded in 2 phases within a hospital intranet. In phase 1, a single-LLM RAG prototype (Qwen3.6-35B-A3B) indexing 6 Japanese oncology guidelines and handling rules was built on a consumer workstation (NVIDIA GeForce RTX 5090; approximately US $5000), and latency was benchmarked on 10 cases. In phase 2, a 5-agent workflow (CrewAI; Gemma 4 26B-A4B on an NVIDIA GB10 Grace Blackwell Superchip) structured the clinical context, extracted clinical issues and missing data, retrieved local guideline and regulatory documents, sent only non-identifying queries to PubMed, and synthesized ranked, source-linked candidates. Reports for 10 consecutive de-identified oral and salivary gland malignancy cases (median age 63, range 41-88 years), some of which had also been used during development, were independently rated by 3 oral and maxillofacial surgeons (24, 14, and 7 years of experience) for treatment-option sufficiency and treatment-planning utility (0-3; score ≥2 prespecified as sufficient or useful) and audited for errors and hallucinations; agreement was quantified by percentage agreement and Fleiss κ.

Results:

In phase 1, the workstation generated approximately 200 tokens/s with a mean end-to-end response time of 4.89 (range 4.15-5.32) seconds per case. In phase 2, the 3 raters assigned identical scores for every case and dimension (Fleiss κ=1.00). Treatment-option sufficiency reached score ≥2 in 10/10 cases (100%) and treatment-planning utility in 9/10 cases (90%). The reports cited 26 unique PubMed-indexed publications, including 4 randomized controlled trials, together with 24 guideline passages and 4 official regulatory documents. The system integrated Japanese guidelines with the KEYNOTE-689 trial and national optimal-use guidance on perioperative pembrolizumab, and flagged a missing biomarker (programmed death-ligand 1 combined positive score) and the uncertainty of extrapolating squamous cell carcinoma trials to adenoid cystic carcinoma. All raters identified 1 prioritization error (1/10, 10%), in which an advanced-stage regimen was ranked first for an early-stage tumor; no fabricated citations were identified by the raters.

Conclusions:

An on-premises multi-agent LLM system combining local guideline RAG with de-identified literature retrieval was implemented, and 3 co-author surgeons rated its reports useful in 9 of 10 cases. Because evaluation cases overlapped with development and half of the reports were corrected before rating, these findings describe a tuned system rather than performance on independent cases; phase 2 responsiveness, privacy protections, and clinical benefit require blinded, comparative assessment before clinical deployment.