Accepted for/Published in: JMIR Medical Education
Date Submitted: Oct 25, 2022
Open Peer Review Period: Oct 24, 2022 - Nov 1, 2022
Date Accepted: Feb 25, 2023
Date Submitted to PubMed: Feb 26, 2023
(closed for review but you can still tweet)
Use of Multiple-Select Multiple-Choice Items in a Dental Undergraduate Curriculum: Retrospective Application of Different Scoring Methods
ABSTRACT
Background:
Scoring and awarding credit on multiple-select items is more complex than on single-choice items. Forty-one different scoring methods were retrospectively applied to two multiple-select multiple-choice item types (Pick-N and Multiple-True-False [MTF] items) from existing exam data.
Objective:
This study aimed to calculate and compare mean scores for both item types by applying different scoring methods, and to investigate the effect of item quality on mean raw scores and the likelihood of resulting scores at or above pass-level (≥0.6).
Methods:
Items and responses from examinees (ie, marking events) were retrieved from previous examinations. Different scoring methods were retrospectively applied to the existing exam data to calculate corresponding examination scores. In addition, item quality was assessed using a validated checklist. Statistical analysis was performed using Kruskal Wallis test, Wilcoxon rank-sum test, and multiple logistic regression analysis (P<.05).
Results:
1,931 marking events on 48 Pick-N items and 828 marking events on 18 MTF items were analysed. For both item types, scoring results widely differed between scoring methods (min: 0.02, max: 0.98, P<.001). Both the use of an inappropriate item type (N = 34 items) and presence of cues (N = 30 items) impacted on the scoring results: Pick-N items used inappropriately resulted in lower mean raw scores (0.88 vs 0.93, P<.001), while inappropriately used MTF items resulted in higher mean raw scores (0.88 vs 0.85, P=.001). MTF items with cues resulted in higher mean raw scores than items without cues (0.91 vs 0.8, P<.001) while mean raw scores from Pick-N items with and without cues did not differ (0.89 vs 0.90, P=.09). Item quality also impacted on the likelihood of resulting scores at or above pass-level (OR≤6.977).
Conclusions:
Educators should pay attention when using multiple-select multiple-choice items and select the most appropriate item type. Different item types, scoring methods, and presence of cues are likely to impact on examinees’ scores and overall examination results.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.