Currently accepted at: JMIR Formative Research
Date Submitted: Nov 24, 2025
Open Peer Review Period: Nov 25, 2025 - Jan 20, 2026
Date Accepted: Aug 4, 2026
(closed for review but you can still tweet)
This paper has been accepted and is currently in production.
It will appear shortly on 10.2196/88404
The final accepted version (not copyedited yet) is in this tab.
Automated Extraction and Coding of Qualitative Data Within Physician-Patient Communication in Cancer Care: Using Natural Language Processing
ABSTRACT
Background:
Large-scale analysis of qualitative clinical communication data typically requires substantial manual coding. Natural Language Processing (NLP) may reduce this burden by converting unstructured physician-patient communication transcripts into structured qualitative data.
Objective:
This study aimed to develop, test, and validate an NLP application for identifying quality-of-life factors in cancer care communication. We also compared traditional machine learning, rule-based, transformer-based, and out of box (OoB) large language model (LLM) approaches.
Methods:
We analyzed 434 standardized patient encounter transcripts of physician-patient interactions collected from a prior study. A rule-based SpaCy model, traditional machine learning models, BERT, RoBERTa, and the OoB Qwen3 large language model were evaluated using F1 scores. The highest-performing model was incorporated into a web-based NLP application and further assessed through manual validation.
Results:
Transformer-based models demonstrated the strongest performance. RoBERTa achieved the highest initial F1 score of 83%, outperforming BERT at 80%, the OoB Qwen3 LLM model at 39%, and the rule-based SpaCy baseline at 34%. Manual validation of the RoBERTa model produced an F1 score of 91%. The comparatively low performance of Qwen3 indicated that OoB LLM use alone may not produce reliable results for domain-specific qualitative clinical communication tasks.
Conclusions:
Fine-tuned transformer model (RoBERTa) was better suited to capturing the complexity of patient encounter transcripts than traditional machine learning, rule-based approaches, or an OoB LLM. The findings demonstrate the feasibility of using a manually validated, web-based NLP application trained on gold-standard annotations to transform large-scale physician-patient communication transcripts into structured qualitative data. Hybrid systems combining rule-based precision with the contextual capabilities of transformer models may further improve the identification of longer conversational sequences, including emotional expressions, question-answer exchanges, and multi-topic utterances.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.