JMIR Publications

ABSTRACT

Background:

Anxiety problems are prevalent among autistic children and adolescents and those with attention-deficit/hyperactivity disorder (ADHD). Natural language processing (NLP) is increasingly used to structure information from electronic health records, but implementation within secure child and adolescent mental health services (CAMHS) research environments is less frequently reported.

Objective:

This study documented the specification, governance, integration, and evaluation of an NLP-enabled information-extraction workflow within Oxford Health NHS Foundation Trust CAMHS research on documented anxiety-treatment information and outcome-related text in autistic and ADHD youth in electronic health records (EHRs) and examined implementation using Normalisation Process Theory (NPT).

Methods:

A multidisciplinary team developed a locally deployed NLP pipeline to identify labelled text spans describing presenting problems, medication attributes, psychological therapies, and treatment outcomes in pseudonymised electronic health record notes. Implementation followed an iterative seven-phase roadmap: (1) defining research questions; (2) identifying key constructs; (3) co-developing an annotation schema; (4) training human annotators; (5) resolving annotation discrepancies; (6) large-scale annotation of EHR data; and (7) NLP model development and evaluation. NPT was applied throughout to guide implementation and evaluate progress. Quantitative outcomes (data volume processed, annotation reliability, and model performance) and qualitative feedback were collected to assess implementation success.

Results:

The source cohort comprised 454,127 notes from 5,906 patients; 697 randomly selected, manually annotated notes from 302 patients were used for model development and held-out evaluation. Category-specific position-based Krippendorff’s alpha ranged from 0.325 to 0.579 for problem, psychological-therapy, and outcome spans and from 0.665 to 0.967 for medication spans. Agreement was highest for medication names, doses, and frequencies and lowest for cognitive problems and treatment outcomes. Held-out macro-averaged F1 was 0.87 (95% CI 0.86-0.89) across the four medication labels and 0.71 (95% CI 0.68-0.73) across the five problem labels, psychological therapy, and outcomes; label-specific F1 ranged from 0.60 to 0.98. Team-generated implementation records indicated that multidisciplinary schema development and clear protocols supported shared understanding and engagement, whereas annotation burden, documentation variability, data-access delays, and computing constraints impeded implementation.

Conclusions:

This report demonstrates the feasibility of integrating NLP tools into NHS CAMHS research workflows to extract structured information on anxiety treatments from routine clinical records. The NPT framework provided a structured framework for documenting the organisational work, barriers, and adaptations involved. External validation, refinement of less reliable labels, and further evaluation are required before broader use.