Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Currently accepted at: JMIR Formative Research

Date Submitted: Nov 24, 2025
Open Peer Review Period: Nov 25, 2025 - Jan 20, 2026
Date Accepted: Aug 4, 2026
(closed for review but you can still tweet)

This paper has been accepted and is currently in production.

It will appear shortly on 10.2196/88404

The final accepted version (not copyedited yet) is in this tab.

Automated Extraction and Coding of Qualitative Data Within Physician-Patient Communication in Cancer Care: Using Natural Language Processing

  • Jay Sureshbhai Patel; 
  • Laura A. Siminoff; 
  • Zasim Azhar Siddiqui; 
  • Maria D. Thomson; 
  • Ritwik Katiyar; 
  • Oqba Memar

ABSTRACT

Background:

Large-scale analysis of qualitative clinical communication data typically requires substantial manual coding. Natural Language Processing (NLP) may reduce this burden by converting unstructured physician-patient communication transcripts into structured qualitative data.

Objective:

This study aimed to develop, test, and validate an NLP application for identifying quality-of-life factors in cancer care communication. We also compared traditional machine learning, rule-based, transformer-based, and out of box (OoB) large language model (LLM) approaches.

Methods:

We analyzed 434 standardized patient encounter transcripts of physician-patient interactions collected from a prior study. A rule-based SpaCy model, traditional machine learning models, BERT, RoBERTa, and the OoB Qwen3 large language model were evaluated using F1 scores. The highest-performing model was incorporated into a web-based NLP application and further assessed through manual validation.

Results:

Transformer-based models demonstrated the strongest performance. RoBERTa achieved the highest initial F1 score of 83%, outperforming BERT at 80%, the OoB Qwen3 LLM model at 39%, and the rule-based SpaCy baseline at 34%. Manual validation of the RoBERTa model produced an F1 score of 91%. The comparatively low performance of Qwen3 indicated that OoB LLM use alone may not produce reliable results for domain-specific qualitative clinical communication tasks.

Conclusions:

Fine-tuned transformer model (RoBERTa) was better suited to capturing the complexity of patient encounter transcripts than traditional machine learning, rule-based approaches, or an OoB LLM. The findings demonstrate the feasibility of using a manually validated, web-based NLP application trained on gold-standard annotations to transform large-scale physician-patient communication transcripts into structured qualitative data. Hybrid systems combining rule-based precision with the contextual capabilities of transformer models may further improve the identification of longer conversational sequences, including emotional expressions, question-answer exchanges, and multi-topic utterances.


 Citation

Please cite as:

Patel JS, Siminoff LA, Siddiqui ZA, Thomson MD, Katiyar R, Memar O

Automated Extraction and Coding of Qualitative Data Within Physician-Patient Communication in Cancer Care: Using Natural Language Processing

JMIR Formative Research. 04/08/2026:88404 (forthcoming/in press)

DOI: 10.2196/88404

URL: https://preprints.jmir.org/preprint/88404

Download PDF


Request queued. Please wait while the file is being generated. It may take some time.

© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.