JMIR Publications

ABSTRACT

Background:

Identifying and selecting relevant scientific evidence is central to evidence-based medicine but challenging to teach early in medical training. Educational interventions have shown inconsistent effects on objectively measured performance, leaving limited evidence on how to improve students’ practical evidence retrieval and selection.

Objective:

To evaluate whether adding a brief introduction to citation searching to standard literature-searching instruction improves the relevance and methodological quality of evidence selected by undergraduate medical students.

Methods:

We conducted a pragmatic controlled quasi-experimental study among second-year medical students in Lille, France. Two pre-existing teaching sessions received the same 45-minute standard literature-searching course. Control students completed the study assessment before, and intervention students after, an additional 15-minute introduction to citation searching, including its automation using BibliZap. BibliZap is an open-source tool for automated multilevel forward and backward citation searching. Students searched 1 of 6 randomly assigned clinical questions for 30 minutes and submitted up to 10 references; BibliZap use was optional and tracked. Two blinded experts independently evaluated academic articles. The primary outcome was the rate of expert top picks, defined as references experts considered particularly important for the assigned question; secondary outcomes assessed relevance and quality. Intention-to-teach analyses compared assigned groups irrespective of BibliZap use; per-use analyses compared controls with intervention students who used BibliZap. Models accounted for the number of references submitted and were adjusted for prespecified participant characteristics.

Results:

Among 450 participating students (79.1% of the class), 222 were controls and 228 intervention students; 71 (31%) intervention students used BibliZap. Median search duration was 21 minutes in both groups (Q1-Q3, 14-28 and 15-27), and the median number of references submitted was 7 (Q1-Q3, 6-8 and 6-9, respectively). In intention-to-teach analyses, citation-searching training increased expert top picks by 14% (incidence rate ratio [IRR] 1.14, 95% CI 1.04-1.25; P=.006), articles experts would include in a talk (IRR 1.10, 95% CI 1.03-1.18; P=.003), and mean relevance grade (+0.30/10, 95% CI 0.20-0.40; P<.001). For expert top picks, the IRR was 1.09 (95% CI 0.98-1.21) with no tracked BibliZap use, 1.24 (95% CI 1.09-1.41) with at least 1 seed, and 1.37 (95% CI 1.14-1.66) with at least 2 seeds. The intervention also reduced nonacademic references by 51% (IRR 0.49, 95% CI 0.33-0.74) and students selected articles with greater methodological rigor (proportional odds ratio 1.42, 95% CI 1.16-1.74; both P<.001).

Conclusions:

Adding a brief introduction to citation searching to standard literature-searching instruction improved the relevance and quality of evidence selected by undergraduate medical students. These findings support citation searching as a practical component of early literature-searching education and warrant evaluation of whether its benefits persist and generalize beyond a single teaching session.