JMIR Publications

ABSTRACT

Background:

Field notes are widely adopted across Canadian medical residency training programs for documenting resident performance feedback. This process generates a large cumulative collection of feedback text, which is difficult for medical education faculty to navigate. As sentiment analysis is a sub-field of text mining that can efficiently synthesize the polarity of a collection of text, sentiment analysis may serve as an innovative solution.

Objective:

To examine the utility and feasibility of sentiment analysis using three popular sentiment lexicons on medical resident field notes.

Methods:

We used a retrospective cohort design, curating text data from University of Toronto medical resident field note data gathered over a two-year period (July 2019 – June 2021). Lexicon-based sentiment analysis was applied using three standardized dictionaries, modified with the removal of ambiguous words as determined by a medical subject matter expert. Our modified lexicons assigned words from the text data a sentiment score, and aggregated word-level scores to a document-level polarity score. Agreement between dictionaries was assessed, and the document-level polarity was correlated with the overall preceptor rating of the clinical encounter under assessment.

Results:

Across the three original dictionaries, around a third of labeled words in our field note corpus were deemed ambiguous and removed to create modified dictionaries. Across all three modified dictionaries, the mean sentiment for the field note “Strengths” section was mildly positive, and slightly less positive in the “Areas of Improvement” section. We observed reasonable agreement between dictionaries for sentiment scores in both field note sections. Overall, the proportion of positively-labeled documents increased with the overall preceptor rating, and the proportion of negatively-labeled documents decreased with the overall preceptor rating.

Conclusions:

Applying sentiment analysis to systematically analyze field notes is feasible. However, the applicability of existing lexicons is limited in the medical setting, even after the removal of ambiguous words. This warrants the need to generate new dictionaries specific to the context of medical education. Additionally, aspect-based sentiment analysis may be applied to navigate the more nuanced structure of texts when identifying sentiments. Ultimately, this will allow for stronger inferences shedding light on opportunities to advance resident teaching curriculums.