JMIR Publications

ABSTRACT

Background:

As artificial intelligence (AI) becomes embedded in health sciences education and clinical practice, understanding the AI literacy of teaching faculty has emerged as a pressing institutional priority. Existing instruments were largely developed for students or general adult populations, leaving limited validation evidence for tools tailored to health sciences educators.

Objective:

This study reports psychometric findings from an AI literacy survey administered to teaching faculty members in health sciences education at a university in Hong Kong. Specifically, it (1) describes the empirically derived factor structure of a 44-item generative AI literacy instrument and evaluates its coherence against the original four-domain theoretical model; (2) assesses the reliability and subscale score profiles of the empirically derived factors to characterize the current AI literacy levels of health sciences teaching faculty; and (3) discusses the implications of these findings for targeted professional development and institutional AI governance, with specific reference to the health sciences education context and identifies future research directions for further psychometric validation.

Methods:

A cross-sectional census-approach survey was administered to teaching faculty members involved in health sciences education at a university in Hong Kong. A 52-item instrument underwent expert review and cognitive interviewing, yielding a refined 44-item version. Exploratory factor analysis with maximum likelihood extraction and varimax rotation examined the preliminary latent structure. Unit-weighted subscale scores were computed, and internal consistency was assessed using Cronbach's α.

Results:

Of 140 eligible teaching faculty members, 126 completed the survey (response rate 90%). Five interpretable factors emerged: Foundational AI Knowledge and Basic Operational Skills; Pedagogical Integration and Classroom Application; Institutional and Collaborative AI Governance; Prompt Engineering and Output Quality Assurance; and Personal Ethical Stance and Responsible Practice. The 44-item pool showed excellent internal consistency (Cronbach's α = 0.965), and empirical subscales showed acceptable to good internal consistency (α = 0.78 - 0.85). Staff reported the strongest scores on Prompt Engineering (mean 7.45 out of 10, SD 1.70) and Personal Ethical Stance (mean 7.15, SD 1.80). Institutional Governance was the only subscale below the midpoint 5.5 (mean 4.35, SD 2.19), indicating a potential collective governance gap.

Conclusions:

AI literacy among health sciences teaching faculty may be more differentiated than the four-domain frameworks anticipate. Our findings distinguish personal ethics from institutional governance and general tool use from advanced prompt engineering. Further multisite validation is required before standardized use. These findings provide an empirically grounded foundation for the design of targeted staff development programs and inform a revised instrument for wider deployment across comparable health sciences education institutions.