JMIR Publications

ABSTRACT

Background:

The Clinical Medicine (Big Data Track) program, an emerging specialty under China's "New Medical Education" initiative, trains hybrid professionals who possess both clinical knowledge and data science competencies. However, internship evaluation for this track remains underdeveloped, relying on traditional time-based attendance sheets or simplified computer science coding assessments that fail to capture the unique competency profile required in real-world medical record data (RWD) settings.

Objective:

This study aimed to construct a comprehensive, quantitative, and competency-oriented evaluation system for assessing internship teaching effectiveness in the Clinical Medicine (Big Data Track), specifically within RWD-based training environments.

Methods:

A three-phase mixed-methods design was adopted. First, an initial indicator pool comprising 4 first-level, 14 second-level, and 42 third-level indicators was developed through literature review and semi-structured interviews with five hospital-based clinical mentors and three university program directors. Second, a two-round modified Delphi survey was conducted with 20 purposively selected experts from higher vocational medical colleges (n=7), tertiary hospitals (n=10), and health informatics enterprises (n=3). Experts rated indicator importance on a 5-point Likert scale and provided qualitative revision suggestions. Third, the Analytic Hierarchy Process (AHP) was applied using pairwise comparison matrices derived from the second-round mean importance scores to determine indicator weights. Consistency was verified (CR < 0.10 threshold).

Results:

Both Delphi rounds achieved a 100% response rate. Expert authority coefficient was 0.87. Kendall's W increased from 0.302 (Round 1) to 0.368 (Round 2) (both P < 0.01), indicating improved consensus. The finalized system comprises 4 first-level, 12 second-level, and 37 third-level indicators. Among first-level indicators, "Data Governance and Technical Application Competency" received the highest weight (0.382), followed by "Medical-Data Integration Cognition and Transformation Competency" (0.296), "Data Professionalism and Ethics" (0.177), and "Project Practice and Decision Support Competency" (0.145). The top five weighted third-level indicators were proficiency in statistical software (SPSS/R/Python) (0.052), using medical knowledge to identify logically conflicting data (0.051), code normalization and coverage rate of data cleaning rules (0.048), SQL proficiency for multi-table queries (0.045), and understanding the underlying EMR data architecture (0.042).

Conclusions:

This Delphi-AHP study yielded a reliable, quantitatively weighted evaluation system emphasizing the dual-core competency structure of technical data capabilities and medical-data integrative cognition. The system addresses the unique requirements of RWD-based training—data ethics safeguards, dirty data governance, and dual-mentor collaborative evaluation—providing a standardized quality assessment tool for the Clinical Medicine (Big Data Track) internship. Empirical validation through multi-institutional field testing represents the next research priority.