JMIR Publications

ABSTRACT

Background:

Large language models (LLMs) are reshaping how medical and dental students acquire knowledge by providing immediate, personalized explanations of unfamiliar concepts. Yet understanding an explanation does not necessarily mean being able to judge its accuracy or reliability. Students may turn to AI precisely when they lack the knowledge needed to evaluate its responses. Much prior research has assessed AI literacy through self-reported knowledge or perceived competence, which may not reflect objectively demonstrable knowledge. Whether trust in LLM outputs reflects objectively assessed GenAI knowledge remains unclear.

Objective:

To investigate whether trust in LLM outputs aligns with objectively assessed GenAI literacy among medical and dental students, and to examine how literacy and trust relate to LLM use and students' evaluations of AI benefits and risks.

Methods:

We conducted a secondary analysis of a multicenter cross-sectional survey of medical and dental students in Germany. GenAI literacy was measured using the 20-item Generative AI Literacy Assessment Test (GLAT), alongside overall trust in LLM outputs and reported use frequency. Adjusted regression analyses examined associations between literacy and trust. Two independent human coders classified open-ended descriptions of GenAI benefits and risks, with disagreements reviewed by a third independent adjudicator. Exploratory analyses examined the number of themes mentioned and the specificity of students' explanations in relation to literacy and trust.

Results:

Among 802 students with paired GLAT and trust data, 752 were included in the primary adjusted analysis. Greater trust in LLM outputs was associated with lower GenAI literacy (adjusted difference, −0.966 GLAT points per trust level; 95% CI −1.347 to −0.585). This inverse association was directionally consistent across all four GLAT knowledge areas, although differences between areas were not statistically conclusive. More frequent LLM use was associated with greater trust (Spearman ρ=0.417; P<.001), but not clearly with GLAT scores (ρ=−0.051; P=.147). Among open-ended responses (579 benefit and 642 risk responses), higher GLAT scores were associated with mentioning more benefit and risk themes and providing more specific explanations in both domains. These associations involved learning support and interaction, answer quality and task fit, and information and evidence risks. Greater trust was associated with mentioning more benefit themes, but showed no clear association with risk theme counts or explanatory specificity.

Conclusions:

Greater trust in LLM outputs was associated with more frequent use but lower objectively assessed GenAI literacy, whereas use frequency itself was not clearly associated with literacy. Higher literacy was associated with broader and more specific evaluations of AI benefits and risks. These findings highlight the importance of fostering knowledge-informed trust in medical education. Objectively assessed GenAI knowledge is an important foundation for educational preparedness, supporting students' understanding of what these systems can and cannot reliably do and their ability to evaluate AI-generated information in context.