JMIR Publications

ABSTRACT

Background:

Competency-based medical education increasingly uses digital workplace assessment platforms to collect longitudinal evidence of trainees’ readiness for progressive responsibility. Entrustable professional activities (EPAs) translate competencies into observable clinical work, but individual workplace observations are context dependent and may not support dependable progression decisions. Empirical guidance remains limited on how many observations, faculty raters, EPA titles, and workplace settings are needed for a sufficient EPA portfolio.

Objective:

This study aimed to estimate EPA portfolio sufficiency for competency-based progression review using nationwide data from a digital workplace assessment platform in otorhinolaryngology–head and neck surgery residency training.

Methods:

We conducted a retrospective, nationwide cross-sectional generalizability and decision study using routinely collected EPA-based workplace assessment data from Taiwan’s E-MyWay platform. The study included completed EPA observations from accredited otorhinolaryngology–head and neck surgery residency programs between August 2022 and May 2026. Faculty-assigned entrustment–supervision ratings were scored on a 5-level national EPA scale. The resident was the object of measurement, and faculty rater, EPA title, and workplace setting were measurement facets. Generalizability analyses decomposed variance in entrustment ratings, and decision-study simulations estimated absolute dependability under alternative portfolio configurations.

Results:

The analytic sample included 45,867 EPA observations involving 466 residents and 448 faculty raters across 36 training programs, 11 EPA titles, and 5 workplace settings. Residents had a median of 88 observations and were assessed by a median of 9 faculty raters across 11 EPA titles and 5 workplace settings. Single EPA observations were insufficient for resident-level summative interpretation. In the primary generalizability study, residual variance was the largest component, accounting for 40.4% of total variance; faculty-rater variance accounted for 10.9%, and resident–faculty rater interaction accounted for 10.5%. A well-distributed 10-observation portfolio achieved a generalizability coefficient of 0.77 but an absolute dependability coefficient (Phi) of 0.65. The first scenario reaching minimally acceptable absolute dependability was 20 observations across 5 faculty raters, 5 EPA titles, and 3 workplace settings (Phi=0.76). For higher-stakes progression review, 30 observations across 6 faculty raters, 6 EPA titles, and 4 workplace settings achieved Phi=0.81.

Conclusions:

Nationwide digital workplace assessment data showed that dependable EPA-based progression review requires both sufficient observation counts and purposeful sampling across faculty raters, EPA titles, and workplace settings. These study-derived benchmarks may help clinical competency committees, residency programs, and specialty societies monitor portfolio completeness before progression deliberation. They should complement, rather than replace, narrative feedback, case complexity, performance trajectory, and committee judgment. Clinical Trial: Not applicable.