JMIR Publications

ABSTRACT

Background:

The negative psychosocial impacts of cancer diagnoses and treatments are well documented. Virtual care has become an essential mode of care delivery during the COVID-19 pandemic and online support groups (OSGs) are shown to improve accessibility to psychosocial and supportive care. The de Souza Institute offers CancerChatCanada, a therapist-led OSG service where sessions are monitored by an artificial intelligence-based co-facilitator (AICF). AICF is equipped with a recommender system that uses natural language processing to tailor online resources to patients according to their psychosocial needs.

Objective:

To outline the development protocol and to evaluate AICF on its precision and recall in recommending resources to cancer OSG members.

Methods:

Human input informed the design and evaluation on its ability to 1) appropriately identify key words indicating a psychosocial concern and 2) recommend the most appropriate online resource to the OSG member expressing each concern. Three rounds of human evaluation and algorithm improvement were performed iteratively.

Results:

We evaluated 7,190 outputs and achieved .797 precision, .981 recall, and an F1 score of .880 by the third round of evaluation. Resources were recommended to 48 patients and 25 (52.1%) accessed at least one resource. Of those who accessed the resources, 75.4% found them useful.

Conclusions:

The preliminary findings suggest that AICF can help provide tailored support for cancer OSG members with high precision, recall, and satisfaction. AICF has undergone rigorous human evaluation and the results provide much-needed evidence, while outlining potential strengths and weaknesses for future applications in supportive care.