JMIR Publications

ABSTRACT

Background:

Patient satisfaction management directly reflects the quality and efficiency of hospital service, which has been an important indicator of the level of modern hospital management. Accurate classification of patient complaints is an important aspect of patient satisfaction management. Traditional subjective classification methods based on manual assessment suffer from low efficiency and accuracy. Therefore, there is a growing need for more advanced and automated approaches for categorizing complaints.

Objective:

In this article we aim to development and validate an intelligent system for classifying patient complaints using machine learning and natural language processing.

Methods:

In this article, we proposed an artificial intelligence-based natural language processing technology to extract the dissatisfactory words with high frequency, including departments, staff, and key treatment procedures. A total of 1,465 complaint data from 2019 to 2023 were selected and semantically analyzed, and all complaints were divided into four types: Communication problem, Diagnosis and treatment, Management problem and Sense of responsibility. First, all data will be used for a semantic analysis of the main distribution of patient complaints in our hospital. This will help us to address specific areas of patient complaints and focus on improving our staff and work processes. Second, 80% of this data is used for training the automatic classification model for complaints, and 20% of the data is used for model validation. The unbalanced data were balanced by oversampling using the SMOTE algorithm to achieve a balanced state for all four categories. In addition, 376 data sets from Hangzhou Cancer Hospital are used for external testing of the discomfort classification model. Three machine learning methods (Multifactor_Logistic Regression, Multinomial_Naive Bayes and Support Vector Machines) were used to train and validate the model.

Results:

The original data consists of 719, 376, 260 and 86 records for communication problems, Diagnosis and treatment, management problems and sense of responsibility, respectively. The Multifactor Logistic Regression and Support Vector Machines achieved a weighted average accuracy of 0.89 and 0.93 in the training set and 0.83 and 0.87 in the internal test set, respectively. The Support Vector Machines algorithm, which performed best in prediction, achieved an average accuracy of 0.91 on the external test set, with a 95% confidence interval of [0.87, 0.97].

Conclusions:

The Natural Language Processing based support vector machine algorithm shows good classification performance on automatic categorization of patient complaint texts. Moreover, it has the best performance in both internal and external test sets. This automatic classification system for patient complaint based on machine learning can greatly increase classification efficiency, reduce manpower and improve classification accuracy. It has great prospects for application in medical institutions with a large number of complaints and a shortage of specialists to handle patient complaints.