JMIR Publications

ABSTRACT

Quality monitoring is essential in psychotherapy clinical trials and for supporting evidence-based interventions in mental health service settings. Observational, moment-to-moment coding is ideally suited for capturing information about the quality of behavioral treatments but is too burdensome in most settings. Natural Language Processing (NLP) has the potential to reduce burden by automating quality coding. Recent work has demonstrated the viability of this approach, but studies have yet to test whether it can be used to capture interactions in traditional, conversation-based therapy or with underrepresented subgroups. To fill these gaps, we describe methods and work to date for testing NLP-based quality coding using audio session recordings from six clinical trials of exposure-based CBT for youth and/or adults with anxiety and related disorders (ARDs), across research and community settings. We describe methods for (1) human coders using a validated, moment-to-moment observational quality measure, (2) the NLP pipeline including training, validation, and testing, (3) mitigating against bias, and (4) data analysis. We also detail plans for engaging a variety of partners (e.g., providers, agency administrators, payers, technology developers) from multiple settings (e.g., community clinic, hospital, industry) to guide the project and enhance future utility. In addition to implications for measuring quality in exposure-based CBT, methods may serve as a framework for automating other quality measures and for early engagement of partners/end-users during AI development. Not applicable