JMIR Publications

ABSTRACT

Background:

When a pharmacy student asks a large language model (LLM) tool a question, it returns an answer and usually an explanation of why, and the student learns from the explanation. A correct answer can sit on a wrong explanation, a response’s reliability is rarely apparent from reading it, and students have no dependable way to tell.

Objective:

We tested whether natural language processing of LLM responses can identify wrong answers and whether a language model rater can reproduce faculty judgments of explanation quality, then applied these methods to Liner Scholar, ChatGPT, and Gemini on pharmacy questions across accuracy, stability, explanation quality, and response style.

Methods:

Each product answered 248 investigator-developed North American Pharmacist Licensure Examination-style questions (744 responses), and two experiments on 24 adapted questions added 792 responses. We scored benchmark responses against the answer key and compared products using exact McNemar tests. Fixed linguistic rules measured length, source references, hedging, boosters, distractor coverage, and readability, which were related to correctness and to faculty ratings in regression models, keyness analysis, and a cross validated text classifier. Four faculty had rated 111 responses, and a language model blinded to product rated all 744 on the faculty codebook and was validated against those ratings.

Results:

The three LLM tools answered 210 of 248 questions correctly together and missed 8 together, with accuracy from 87.5% to 95.6%, so differences rested on 30 questions. In every product, wrong answers were longer and named a source more often, and a response naming a source had 0.41 times the odds of being correct. A classifier using only the response, without the question, identified wrong answers with an area under the curve of 0.79. The model rater agreed with faculty as closely as faculty agreed with each other (weighted kappa 0.53 to 0.77 against 0.51 to 0.67 on the three ordinal variables), extending review to 744 responses. Across those responses, 63 of 680 correct answers (9.3%) carried a rationale with a meaningful error and 42 (6.2%) were rated educationally poor, and accuracy and explanation quality ranked the tools in different orders.

Conclusions:

The text of an LLM response carries measurable evidence of its own reliability. Fixed linguistic rules and a classifier identified the answers most likely to be wrong, and a model rater reproduced faculty judgment on rationale correctness, educational quality, and clinical safety across every response, so explanation quality can be audited at scale. The approach needs only response text and a rubric, so any health professions program can apply it to the tools students use, and evaluation of these tools should score the explanation as well as the answer.