JMIR Publications

ABSTRACT

Background:

General-purpose large language models (LLMs) such as free-tier ChatGPT are increasingly used by medical students in Vietnam, yet they lack the domain specificity, Socratic pedagogical structure, and pharmacological safety mechanisms required for Traditional Vietnamese Medicine (TVM) education. TVM presents distinctive epistemological challenges — including dialectical clinical reasoning (biện chứng luận trị), classical herb incompatibilities (Thập bát phản, Thập cửu úy), and a knowledge framework grounded in Vietnamese classical texts — that general-purpose AI systems face fundamental limitations in addressing. No validated AI teaching assistant specifically designed for TVM education has been previously reported.

Objective:

This study aimed to (1) develop TLAS-YHCT VN, a Socratic AI teaching assistant for TVM built on a Retrieval-Augmented Generation (RAG) architecture with multi-layer pharmacological safety guardrails; (2) iteratively refine and validate the system through a four-round human-in-the-loop (HITL) expert process; (3) evaluate student-perceived quality and acceptance compared with free-tier ChatGPT across 30 criteria; (4) assess safety robustness through adversarial red-teaming; and (5) propose a reusable evaluation instrument and validation framework for AI teaching assistants in traditional medicine education.

Methods:

The study comprised two phases. In Phase 1 (October 2025–January 2026), system quality was assessed and refined across four iterative rounds using a 19-criterion expert framework. Claude Opus 4.8 served as an automated LLM-as-a-Judge evaluator across 3,008 test scenarios, while 12 domain experts performed human-in-the-loop scoring and guided system refinement. Judge reliability was validated against blinded expert scoring on 15,140 score pairs. In Phase 2 (May 2026), a cross-sectional survey was conducted among 320 medical students (Years 1–6) at Hoa Binh University (406 responses collected; 78.8% valid after data cleaning). Students rated both TLAS-YHCT VN and free-tier ChatGPT using a validated 30-item, 5-domain instrument (Cronbach α=.98; EFA confirmed two-factor structure, KMO=0.970). Safety robustness was assessed by five experts across 206 adversarial attack scenarios (1,030 scoring instances). The study received ethics approval (No. 23/HĐĐĐ-VKC, January 26, 2026).

Results:

Phase 1 iterative refinement improved expert-rated quality from 3.79 to 4.49 out of 5 (Cohen's d_z=1.70). LLM-as-a-Judge (No-sim mode) demonstrated agreement within ±1 point in 93.4% of cases (MAE=0.57); expert-simulated mode achieved MAE=0.19. In Phase 2, TLAS-YHCT VN scored significantly higher than free-tier ChatGPT across all 30 criteria (composite: 3.94 vs 3.34; Δ+0.60; all P<.001 after Holm-Bonferroni correction). Largest advantages: curriculum alignment (+0.76), Socratic questioning (+0.71), diagnostic reasoning feedback (+0.71). This advantage was consistent across all academic year subgroups (P>.05). After using TLAS-YHCT VN, 77.8% (249/320) reported using it at least several times per week. In adversarial red-teaming, TLAS-YHCT VN produced no harmful responses across all 206 attack scenarios (95% CI upper bound ≈1.5%, rule of three), outperforming free-tier ChatGPT (88.9% safe) and Gemini Flash (78.7% safe).

Conclusions:

TLAS-YHCT VN is, to the authors' knowledge, the first reported RAG-based Socratic AI teaching assistant purpose-built for Traditional Vietnamese Medicine education. It achieved higher domain accuracy, pedagogical quality, and pharmacological safety ratings compared with general-purpose LLMs. The validated 30-item instrument and LLM-as-a-Judge validation framework are proposed as transferable tools for quality assurance of AI teaching assistants in traditional medicine education settings.