JMIR Publications

ABSTRACT

Background:

Gastrointestinal (GI) disorders affect a substantial proportion of the Indonesian population, while access to specialist care remains limited and uneven. Large language models (LLMs) may support initial GI consultation, but evidence on Indonesian-language LLM triage remains limited.

Objective:

This study evaluated Ailix: GI Triage by examining MedikoGPT (MQ), a structured rule-based pathway, with GPT-4o (Convo), a free-conversation pathway, against pathway-specific physician assessment and comparing user-experience and preference.

Methods:

This single-center, observational, cross-sectional study used a within-subject design. Adults with a history of GI complaints completed both pathways in one online session, followed by a study-specific user-experience and preference questionnaire. A physician retrospectively assigned red-flag classifications and provisional diagnoses from each pathway’s consultation records before viewing AI conclusions, then rated output quality without blinding to pathway. These assessments provided pathway-specific comparators rather than a shared independent clinical reference standard. No physical examination, laboratory testing, or imaging was performed. No physical examination or confirmatory testing was performed. Analyses included descriptive alarm-flag cross-tabulations, diagnostic matching with 95% CIs, Gwet AC1, paired Wilcoxon signed-rank tests, binomial preference tests, and thematic analysis. Holm adjustment was applied separately to reviewer-score and user-experience comparisons.

Results:

Among 102 participants, MQ recorded an alarm flag in 39 of 40 physician-classified red-flag cases. Convo recorded “No” in all consultations, but had no equivalent questionnaire-triggered flagging mechanism. Top-1 diagnosis match was 44.1% for MQ and 39.2% for Convo. Top-3 diagnosis match was 81.4% (95% 72.4–88.4) for MQ and 63.7% (95% CI 53.6–73.0) for Convo, while Top-3 diagnosis-set agreement was 25.5% and 22.5%, respectively. Diagnosis-group AC1 was 0.615 and 0.533. These estimates used separate pathway-specific comparators. These estimates referred to separate pathway-specific comparators. MQ received higher safety-net ratings (median 2 [IQR 2–2] vs 2 [IQR 1–2]; Holm-adjusted P<.001), while Convo received higher reviewer-rated clinical accuracy scores (2 [IQR 2–3] vs 2 [IQR 1–2]; Holm-adjusted P=.02). Total reviewer and user-experience scores did not differ significantly after adjustment. Participants preferred Convo for helpfulness (67/102, 65.7%), trustworthiness (65/102, 63.7%), and future use (71/102, 69.6%); all three preferences remained significant after adjustment.

Conclusions:

The two configurations showed different patterns across the assessed dimensions. MQ received higher safety-net ratings, while Convo received higher reviewer-rated clinical accuracy scores and was preferred for helpfulness, trustworthiness, and future use. The findings do not establish comparative diagnostic accuracy or clinical safety. These findings support future evaluation of a hybrid model combining structured safety screening with conversational symptom elaboration. Prospective studies are needed to establish clinical safety and effectiveness.