JMIR Publications

ABSTRACT

Background:

Exposure to patients and clinical diagnoses drives learning in graduate medical education (GME). Measuring practice data, how trainees each experience that exposure, is critical to planned learning processes including assessment of trainee needs. We previously developed and validated an automated system to accurately identify resident provider-patient interactions (rPPIs).

Objective:

In this study, we employ human-centered design methods to meet two objectives: 1) understand trainees’ planned learning needs; 2) design, build, and validate the usability and utility of a tool based on our automated rPPI system to meet these needs.

Methods:

We collected data at two institutions new to the American Medical Association’s “Advancing Change” initiative, using a mixed-methods approach with purposive sampling. First, interviews and formative prototype testing yielded qualitative data which we analyzed with several coding cycles. Interview guides were built to collect data required for a work domain assessment, learning use case elicitation, and ultimately design requirement identification. Two theoretical models—the Systems Engineering Initiative for Patient Safety (SEIPS) and Master-Adaptive Learner (MAL) —structured coding efforts. Feature-prioritization matrix analysis then transformed qualitative analysis outputs into actionable prototype elements that were refined through formative usability methods. Lastly, qualitative data from a summative usability test validated the final prototype with measures of usefulness, usability, and intent to use. Quantitative methods (time on task and task completion rate) were employed in summative testing.

Results:

We represent GME work domain assessment through process-map-design artifacts which provide target opportunities for intervention. Of the identified decision-making opportunities, trainee-mentor meetings stood out as optimal for delivering reliable practice-area information. We designed a “mid-point” report for the use case of such meetings. We arrived at a final prototype through formative testing and design iteration. This final version showed five essential visualizations. Summative usability testing resulted in high performance in subjective and objective metrics. Compared to currently available resources, our tool scored 50% higher in terms of Perceived Usability and 60% higher on Perceived Ease of Use.

Conclusions:

We describe the multi-site development of a tool providing visualizations of log level electronic health record data, using human-centered design methods. Delivered at an identified point in graduate medical education, the tool is ideal for fostering the development of master adaptive learners. The resulting prototype is validated with high performance on a summative usability test. Additionally, the design, development, and assessment process may be applied to other tools and topics within clinical informatics.