JMIR Publications

ABSTRACT

Background:

SMS text message programs are increasingly being used for secondary prevention, and have been shown to be effective in a number of health conditions including cardiovascular disease. Text programs have the potential to increase the reach of an intervention to larger numbers of people who may not access traditional programs, at an affordable cost to the health system. However, patients regularly reply to the text messages, leading to additional staffing requirements to monitor and moderate the patients’ text replies. This additional staff requirement directly impacts the cost effectiveness and scalability of text interventions.

Objective:

This study aimed to test the feasibility and accuracy of developing a machine learning (ML) program to ‘triage’ text message replies (i.e. identify which text message replies require a health professional review).

Methods:

Text message replies received from two clinical trials were manually coded firstly into ‘is staff review required?’ (binary response of yes/no); and secondly into 12-general categories. Five ML models (‘Naïve Bayes’, ‘OneVsRest’, ‘Random Forest decision trees’, ‘Gradient Boosted Trees’, ‘Multi-Layer Perceptron’) and an ensemble model were tested. For each model-run, data were randomly allocated into training-set (70%) and test-set (30%). Accuracy for yes/no classification was calculated using area under curve [AUC], false positives, and false negatives. Accuracy for classification into 12-categories was compared using multi-class classification evaluators.

Results:

Manual review of 3,118 text replies showed 22% required staff review. For determining need for staff review, the Multi-Layer Perceptron model had highest accuracy (area under curve [AUC] 0.86; 4.85% false negatives; 4.63% false positives); with addition of ‘heuristics’ (specified keywords) fewer false negatives were identified (3.19%) with small increase in false positives (7.66%) and AUC 0.79. Application of this model would result in 26.7% of text replies requiring review (true + false positives). The ensemble model produced the lowest false negatives (1.43%) at expense of higher false positives (16.19%). OneVsRest was most accurate (72.3%) for the 12-category classification.

Conclusions:

The ML program has high sensitivity for identifying the text replies requiring staff input. Incorporation of a ML program to review text replies could significantly reduce staff workload, as staff would not have to review all incoming texts. This could lead to substantial improvements in cost-effectiveness, scalability and capacity of text-based interventions.