JMIR Publications

ABSTRACT

Background:

Translating evidence-based therapies from "bench to bedside" remains challenging and implementation Science (IS) experts are crucial for this process. Qualitative analyses are essential, but require extensive time and cost for manual coding. Now, many turn to Artificial Intelligence (AI) to accelerate the pace of qualitative analysis, but significant questions remain about the quality, validity, and ethics of applying Large Language Models (LLMs) like ChatGPT to qualitative data. To this end, we have developed a method for AI-assisted rapid qualitative analysis that addresses these concerns.

Objective:

We developed AI Assisted Rapid Qualitative Analysis for Implementation Science (AI-RQA) as an open-source encoder-based Small Language Models (SLM) to aid IS experts. SLMs hold advantages over LLMs in terms of processing efficiency, customizable training, and local maintenance of data that never goes “to the cloud”. We focus on two efficient and high-performing SLMs – DistilBERT and ELECTRA. The objective is to assess these models’ accuracy in reproducing expert coding, their generalizability to new coding scenarios, and enhancing their accessibility for non-technical experts through user-friendly tools.

Methods:

Two previously coded IS datasets were used to train DistilBERT and ELECTRA models. These datasets were coded by IS experts using a mixed deductive and inductive approach, with initial categories derived from the domains of an IS framework, PRISM. We fine-tuned and evaluated DistilBERT and ELECTRA on these datasets, measuring performance by AUPR and Cohen’s Kappa. To facilitate use by non-programmers, we then developed an open-source Python package (pytranscripts) to streamline transcript processing, model classification, and evaluation. Additionally, a companion Streamlit web application allows users to upload interview transcripts and obtain automated coding and analytics without any coding expertise.

Results:

Our findings demonstrate the success of leveraging SMLs to significantly accelerate qualitative analysis while maintaining high levels of accuracy and agreement with human annotators, though results are not universal and depend on how researchers approach qualitative coding. On the original dataset, DistilBERT achieved near-perfect agreement with human coders (Cohen’s Kappa = 0.95), while ELECTRA showed substantial agreement (Cohen’s Kappa = 0.71). However, both models’ performance declined on the second, more ambiguous dataset, with DistilBERT’s Cohen’s Kappa dropping to 0.48 and ELECTRA’s to 0.39. Two primary drivers of performance drop appear to be related to the number of codes applied to the dataset, and if coders apply multiple codes to each piece of data, or constrain themselves to applying one.

Conclusions:

This work demonstrates that SLMs can meaningfully assist qualitative researchers with coding tasks as long as attention is paid to how experts code data that will train the SLM. This can be especially valuable in settings where deploying LLMs is impractical or undesirable.