JMIR Publications

ABSTRACT

Background:

Administrative costs in health care system are driven in part by complex insurance eligibility determination, particularly for anticancer drugs with multiple AND/OR reimbursement conditions. Large language models (LLMs) are increasingly used for health insurance–related queries, yet their reliability for structured logical reasoning over coverage criteria has not been systematically evaluated.

Objective:

This study aimed to develop a benchmark for anticancer drug reimbursement eligibility determination and evaluate whether LLMs can reliably perform eligibility verification.

Methods:

We constructed a benchmark based on South Korea's National Health Insurance reimbursement guidelines for 3 gynecologic cancers (cervical, uterine, and ovarian), covering 74 anticancer regimens with 222 verification cases (eligible, ineligible, and undeterminable). Eight LLMs from 4 providers were evaluated with the guideline PDF provided directly. Accuracy was the primary outcome, with 95% CIs estimated by the Wilson method. Qualitative error analysis was performed on model-generated rationale text.

Results:

Overall verification accuracy ranged from 57.2% to 82.4% across the 8 models. Eligible and ineligible cases were classified with high accuracy, but undeterminable cases showed a marked decline across all models (12.2%–62.2%). Models predominantly misclassified undeterminable cases as eligible rather than ineligible. Error analysis of 471 incorrect predictions identified 3 failure modes: information gap-filling (60.9%), criterion misapplication (16.3%), and false uncertainty (7.2%). Performance varied by cancer type, with uterine cancer showing the lowest undeterminable accuracy (18.4%), corresponding to the highest guideline complexity.

Conclusions:

Current LLMs cannot reliably perform unsupervised anticancer drug reimbursement eligibility determination. Undeterminable cases expose a systematic tendency to infer eligibility from incomplete information. The identified failure modes, particularly information gap-filling, must be addressed before clinical deployment, though overall accuracy suggests potential as a decision-support tool under human oversight.