JMIR Publications

ABSTRACT

Background:

Objective Structured Clinical Examinations (OSCE) are a useful method to evaluate medical student performance in the clerkship years. OSCEs are designed to assess skills and knowledge in a standardized clinical setting, and via a preset standard grading sheet, so that clinical knowledge can be evaluated at a high level and in a reproducible way.

Objective:

To present our OSCE assessment tool designed specifically for Radiology clerkship medical students, which we called the Objective Structured Radiology Examination, with the intent to advance assessment of clerkship medical students by providing an (1) objective, (2) structured, (3) reproducible and (4) low-cost method to evaluate medical student level Radiology knowledge. To evaluate the reproducibility of this assessment tool.

Methods:

We designed 9 different OSRE examination cases for Radiology clerkship classes with participating 3rd and 4th year medical students. Each examination comprises 1 to 3 images, a clinical scenario and structured questions, along with a standardized scoring sheet that allows for objective and low-cost assessment. Each medical student completed 3 of 9 random examination cases during their rotation. In order to evaluate for reproducibility of our scoring sheet assessment tool, we used 5 examiners to grade the same students. Reproducibility for each case and consistency for each grader were assessed with a two-way mixed effects intra-class correlation (ICC). ICC below 0.4 was deemed poor to fair, an ICC of 0.41-0.60 was moderate, 0.6-0.8 was substantial, and greater than 0.8 was almost perfect. We also assessed correlation of scores and the student’s clinical experience with linear regression model and compared mean grades between 3rd and 4th year students.

Results:

181 students (156 3rd and 25 4th year students) were included in the study, a full academic year. Six of 9 cases demonstrated average ICC more than 0.6 (substantial correlation) and average ICCs ranged from 0.36-0.80 (P < .001 for all of the cases). The average ICC for each grader was more than 0.60 (substantial correlation). The average grade among third years was 11.9 ± 4.9, compared to 12.8 ± 5 among fourth years (P = .005). There was no correlation between clinical experience and OSRE grade (–0.02, P = .48) adjusting for medical school year.

Conclusions:

Our OSRE is a reproducible assessment tool with most of our OSRE cases showing substantial correlation, with the exception of three cases. No expertise in radiology is needed to grade these examinations using our scoring sheet. There was no correlation between scores and clinical experience of the medical students tested.