JMIR Publications

ABSTRACT

Background:

New interns must respond to overnight messages from nurses, yet medical students have limited opportunities to practice this work before residency. Nurse Althea is an artificial intelligence (AI) nurse messaging simulation designed for cross-cover training.

Objective:

We examined implementation, learner survey responses, automated formative scores, and direct operating costs across four medical school sites.

Methods:

From January through May 2026, fourth-year medical students completed simulated overnight cross-cover cases involving nurse messages, a simulated chart, order entry, and documentation. Optional anonymous post-case surveys asked about realism, confidence, and perceived improvement. A large language model applied a physician-authored rubric to generate communication and medical decision-making scores (each 0–5). Analyses were descriptive and exploratory.

Results:

The score dataset contained 3,838 case attempts associated with 400 distinct student identifiers. Students submitted 2,144 completed anonymous surveys, equivalent to 55.9% of scored attempts. Of 2,000 valid realism responses, 1,436 (71.8%) rated a case realistic or very realistic. The mean automated score was 7.62/10. Mean medical decision-making scores exceeded communication scores by 0.79 points (95% CI 0.77–0.82), and 3,316 decision-making scores (86.4%) were 4 or 5, indicating a ceiling. Scores on advanced cases were higher than scores on easy or intermediate cases. Among 395 students with repeated attempts, mean communication and overall scores increased by 0.17 points (95% CI 0.08–0.27) and 0.18 points (95% CI 0.03–0.34), respectively; case mix also changed. Direct platform and model-inference costs were approximately $950, or $0.25 per scored case.

Conclusions:

Nurse Althea supported repeated cross-cover simulation across four sites at a direct operating cost of approximately $0.25 per case. Most valid realism responses rated the cases realistic or very realistic. The formative rubric has not been independently validated, its domains have different scoring rules, and the results do not establish competence or learning gains.