JMIR Publications

ABSTRACT

As large language models (LLMs) increasingly generate health advice, expert human annotation has become central to evaluating safety and constructing training datasets. Yet annotation itself is a noisy, multidimensional measurement task whose challenges remain underexamined. Drawing on our experience developing the MedExpert dataset, an expert-annotated collection of LLM-generated responses in mental health and prenatal care, we identify three core challenges: (1) inter-rater variability driven by underspecified constructs, calibration drift, and heuristic reasoning rather than substantive clinical disagreement; (2) consequential reasoning failures that conventional rubric categories (factuality, completeness) fail to capture, including miscalibrated urgency, misplaced reassurance, and absent triage logic; and (3) recursive interaction between model vulnerabilities and annotation instability. Mental health contexts intensify all three challenges: harm may be relational and behavioral rather than strictly biomedical, tone and framing become safety-relevant variables, and even technically accurate responses can worsen risk by invalidating experience or softening urgency. We propose rubric refinement, shared clinical mental models, and structured calibration as methodological interventions essential for detecting failures and generating robust datasets. Critically, these interventions rest on a conceptualization of expert clinical annotation as a noisy measurement process rather than a means of discovering ground truth. Applying this framing generates practical consequences for interpreting variability, designing rubrics, and building calibration infrastructure.