JMIR Publications

ABSTRACT

Background:

Lexicon-based sentiment analysis of healthcare workforce free-text in French is typically performed by directly applying the FEEL (French Expanded Emotion Lexicon) to a tokenized corpus, with topic structure recovered through Latent Dirichlet Allocation (LDA).

Objective:

On small to medium corpora drawn from clinical settings, this standard pipeline has four known limitations: (i) profession-specific medical acronyms are treated as out-of-lexicon tokens, (ii) French negation operators are ignored, (iii) LDA topic structure is unstable on small text collections, and (iv) inter-rater reliability is rarely quantified.

Methods:

We developed an augmented FEEL-based pipeline that addresses these four limitations through: (1) a 26-entry French emergency medicine acronym expansion dictionary applied prior to lexicon matching, (2) a deterministic French negation handler that flips polarity assignments for tokens within four positions downstream of a negation marker, (3) Non-negative Matrix Factorization (NMF) with TF-IDF input, coherence-based selection of the topic count, and bootstrap stability assessment substituting for LDA, and (4) a dual-reference Cohen's kappa protocol on a stratified 50-response sample, in which one expert human rater and one large language model rater (Claude Opus 4) independently label the same responses, the human rater providing the substantive validation reference and the LLM rater serving as a transparent second annotator following recent methodological work on LLM-augmented text classification benchmarks. We benchmarked the augmented pipeline against a baseline R tidytext topicmodels pipeline on the PERCEPT'urg national survey corpus (n = 141 French emergency medicine professionals, 13 free-text items).

Results:

Acronym expansion added 252 token occurrences to the FEEL-matched set (16 distinct acronyms; SMUR n = 74, SAMU n = 46, IDE n = 34), raising token-level coverage from 39.5% to 40.7% (+1.2 pp). Negation handling triggered 697 polarity reassignments and shifted three items into negative AFINN-style intensity territory that the baseline pipeline classified as net-positive (definition of a non-urgent situation: +0.67 to -0.52; definition of an unscheduled situation: +0.28 to -0.41; ideal medicine in terms of patients: +0.14 \to -0.60). Coherence-based topic-count selection identified k = 3 for NMF (mean C_V = 0.72, versus k = 5 and C_V = 0.54 for the original LDA configuration). NMF was substantially more stable than LDA across re-fits (mean pairwise top-10 Jaccard 0.81 versus 0.48); with deterministic non-negative SVD initialization, within-init NMF stability reached Jaccard= 1.00. The augmented pipeline confirmed and sharpened the substantive finding of a preserved-ideal / dissonant-operational-reality configuration in the PERCEPT'urg corpus, with four rather than one item in negative territory. Dual-reference validation on the stratified 50-response sample yielded human-vs-LLM Cohen's kappa = 0.804 (95% CI [0.639, 0.934]; substantial agreement, 88% raw), establishing that the labeling task is well-defined; lexicon-vs-human kappa = 0.312 ([0.130, 0.493]) and lexicon-vs-LLM kappa = 0.353 ([0.155, 0.559]) were both fair, with the near-equality of the two values indicating that the FEEL pipeline diverges from both references in the same way --- a quantified over-assignment of the positive category driven by bounded lexicon coverage of operationally-negative vocabulary.

Conclusions:

Four targeted augmentations to the standard FEEL pipeline---acronym expansion, negation handling, NMF substitution for LDA, and quantitative inter-rater validation---produce a methodologically defensible workflow for French healthcare workforce free-text without external dependencies beyond FEEL and standard R/Python packages. The framework is transferable to other French clinical corpora and, by replacing the acronym dictionary, to other professional French sub-languages.