JMIR Publications

ABSTRACT

Background:

The promise of artificial intelligence (AI) in medicine depends on its ability to learn from data that reflect what matters to patients and clinicians in the care process. Most existing models are trained on electronic health records (EHRs), which primarily capture biological measures but rarely the interactions and relationships between patients and clinicians. These relationships, central to how care is understood, negotiated, and delivered, unfold across multiple modalities, including voice, text, and video, yet remain largely absent from current datasets. As a result, AI systems trained solely on EHRs risk perpetuating a narrow biomedical view of medicine and overlooking the lived exchanges that define clinical encounters.

Objective:

To design, implement, and evaluate the feasibility of a longitudinal, multimodal system for capturing patient–clinician encounters, linking 360° video/audio recordings with post-visit surveys and EHR data, to create a foundational dataset for downstream AI research.

Methods:

This single-site study is conducted in an academic outpatient specialty clinic (Division of Endocrinology, Mayo Clinic, Rochester, Minnesota). Adult patients attending in-person visits with participating clinicians are invited to enroll. Encounters are recorded with a 360° 2D monocular video camera and dual-channel audio. After each visit, patients complete a brief survey assessing relational empathy, satisfaction, visit pace, and treatment burden. Demographic and clinical data are extracted from the EHR. Feasibility is assessed using five prespecified endpoints: clinician consent, patient consent, recording success, survey completion, and data linkage across modalities.

Results:

Recruitment began in January 2025. By August 2025, 35 of 36 eligible clinicians (97%) and 212 of 281 approached patients (75%) had consented. Of the consented encounters, 162 (76%) resulted in a complete 360° video recording, and 204 patients (96%) completed the post-visit survey. Final feasibility findings are anticipated in early 2026.

Conclusions:

This study aims to demonstrate the feasibility of a replicable framework for capturing the multimodal dynamics of patient–clinician encounters. By detailing workflows, endpoints, and ethical safeguards, it provides a template for generating rich longitudinal datasets. Establishing feasibility lays the foundation for next-generation AI models designed to incorporate the complexity of clinical care.