JMIR Publications

ABSTRACT

Background:

General surgery residency and specialty-training programs face record application volumes, evolving assessment policies (including the exclusion of the US Medical Licensing Examination [USMLE] Step 1 numeric scores, and the rapid emergence of large language models (LLMs) used by both applicants and reviewers. Artificial intelligence (AI) tools have been proposed for applicant screening, personal statement evaluation, letter-of-recommendation analysis, and detection of AI-generated text, raising both efficiency promises and fairness accountability requirements.

Objective:

This narrative review synthesizes peer-reviewed evidence published since 2020 on AI applications in general surgery residency selection and to derive implementation recommendations for general surgery programs.

Methods:

We conducted a targeted, proportionate narrative evidence review of literature from 2020 onward, using keyword concepts spanning artificial intelligence, machine learning, and residency or specialty-training selection, supplemented by hand-searching of reference lists. General surgery was the primary focus with supplementary inclusion of related surgical specialties. Two reviewers independently screened records for relevance and extracted data on sample size, AI method, human comparator, concordance and agreement metrics, predictive performance, selection-criteria weights, and fairness findings. The review was not registered and was not designed as a systematic review.

Results:

Fifteen studies applying AI or machine learning methods to a residency- or fellowship-selection task met inclusion criteria; a seventeenth human-only holistic-review study is discussed as contextual comparator. Five studies provided direct general-surgery evidence and ten addressed adjacent surgical or medical specialties (otolaryngology, orthopedic surgery, plastic surgery, obstetrics and gynecology, and emergency medicine). Three findings were convergent across the evidence base. First, AI-based applicant screening can shift cohort composition relative to human review, with only 7% overlap between AI and program-director selections in the single available general-surgery comparison. Second, large language models achieve only slight agreement with human reviewers when scoring personal statements (weighted kappa 0.12-0.18) and tend to undervalue implicit leadership and resilience narratives. Third, human reviewers cannot reliably distinguish AI-generated from human-written personal statements and letters of recommendation, a pattern replicated across general surgery and adjacent specialties, and perceived authorship influenced interview-offer decisions in at least one study.

Conclusions:

Direct evidence on AI in general surgery residency selection is limited but consistently shows that AI alters selection cohorts, produces low agreement with humans on personal statement scoring, and inherits historical decision patterns. ACGME-accredited programs should treat AI as decision support behind program director accountability, with local validation, prespecified fairness audits, screen-in-only deployment, documented data governance, and post-match outcome monitoring.