JMIR Publications

ABSTRACT

Background:

Medication safety in residential care facilities is a critical concern, particularly when non-medical staff provide medication assistance. The complex nature of medication-related incidents in these settings, coupled with the psychological impact on healthcare providers, underscores the need for effective incident analysis and preventive strategies. Thorough understanding of the root causes, typically through incident-report analysis, is essential for mitigating medication-related incidents.

Objective:

This study aimed to develop and evaluate a multi-label classifier using natural language processing (NLP) to identify the factors contributing to medication-related incidents using incident report descriptions from residential care facilities, with a focus on incidents involving non-medical staff.

Methods:

We analyzed 2,143 incident reports, comprising 7,121 sentences, from residential care facilities in Japan between April 1, 2015, and March 31, 2016. The incident factors were annotated using sentences based on an established organizational factor model and previous research findings. The following nine factors were defined: procedure adherence, medication, resident, resident family, non-medical staff, medical staff, team, environment, and organizational management. To assess the label criteria, two researchers with relevant medical knowledge annotated a subset of 50 reports; the inter-annotator agreement (IAA) was measured using Cohen's kappa. The entire dataset was subsequently annotated by one researcher. Multiple labels were assigned to each sentence. A multi-label classifier was developed using deep learning models, including two Bidirectional Encoder Representations from Transformers (BERT) type models (Tohoku-BERT/UTH-BERT) and an Efficiently Learning an Encoder that Classifies Token Replacements Accurately (ELECTRA), pretrained on Japanese text. Both sentence- and report-level training were performed; the performance was evaluated by the F1 score and exact match accuracy through five-fold cross-validation.

Results:

Among all 7121 sentences, 1167, 694, 2455, 23, 1905, 46, 195, 1104, and 195 included “procedure adherence,” “medicine,” “resident,” “resident family,” “non-medical staff,” “medical staff,” “team,” “environment,” and “organizational management,” respectively. Owing to limited labels, "resident family" and "medical staff" were omitted from the model development process. The IAA values were higher than 0.6 for each label. A total of 10, 278, and 1855 reports contained no, one, and multiple label(s), respectively. The models trained using the report data outperformed those trained using sentences, with macro F1 scores of 0.744, 0.675, and 0.735 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. The report-trained models also demonstrated better exact match accuracy, with 0.411, 0.389, and 0.399 for Tohoku-BERT, UTH-BERT, and ELECTRA, respectively. Notably, the accuracy was consistent even when the analysis was confined to reports containing multiple labels.

Conclusions:

The multi-label classifier developed in our study demonstrated potential for identifying various factors associated with medication-related incidents using incident reports from residential care facilities. Thus, this classifier can facilitate prompt analysis of incident factors, thereby contributing to risk management and development of preventive strategies.