JMIR Publications

ABSTRACT

Background:

Recommender systems help reduce down a large range of items to a smaller, personalized set of items. NarraGive is a first-in-field hybrid recommender system for mental health recovery narratives, recommending narratives based on their topic, tone and narrator characteristics (using content-based filtering) and on narratives beneficially impacting on other people with similar preferences (using collaborative filtering). NarraGive is integrated into the NEON Intervention, a web application providing access to the NEON Collection of recovery narratives.

Objective:

To evaluate NarraGive by describing how participants used it and by comparing the performance of the content-based filtering algorithm with the two collaborative filtering algorithms.

Methods:

Using a recently-published framework for evaluating recommender systems to structure the analysis, we compared the content-based filtering algorithm and collaborative filtering algorithms by evaluating the accuracy (how close the predicted ratings are to the true ratings), precision (the proportion of the recommended narratives that are relevant), diversity (how diverse the recommended narratives are), coverage (the proportion of all available narratives that can be recommended), and unfairness (whether the algorithms produce less accurate predictions for disadvantaged participants) across gender and ethnicity. We used data from all participants in two parallel group, wait list control clinical trials of the NEON Intervention (NEON Trial: ISRCTN11152837, n=739; NEON-O Trial: ISRCTN63197153, n=1023). Both trials included people with self-reported mental health problems who had and had not used statutory mental health services. Additionally, NEON Trial participants had experienced self-reported psychosis in the previous five years. Our evaluation utilized a database of Likert scale narrative ratings provided by trial participants, in response to validated narrative feedback questions.

Results:

Participants from the NEON Trial and NEON-O Trial provided 2288 and 1896 narrative ratings respectively. Each rated narrative had a median of 3 ratings and 2 ratings respectively. For the NEON Trial, the content-based filtering algorithm performed better for coverage, the collaborative filtering algorithms performed better for accuracy, diversity, and unfairness across both gender and ethnicity, and neither algorithm performed better for precision. For the NEON-O Trial, the content-based filtering algorithm did not perform better on any metric, the collaborative filtering algorithms performed better on accuracy and unfairness across both gender and ethnicity, and neither algorithm performed better for precision, diversity or coverage.

Conclusions:

Clinical population may be associated with recommender system performance. Recommender systems are susceptible to a wide range of undesirable biases. Approaches to mitigating these include providing enough initial data for the recommender system (to prevent overfitting), ensuring items can be accessed outside the recommender system (to prevent a feedback loop between accessed items and recommended items), and encouraging participants to provide feedback on every narrative they interact with (to prevent participants from only providing feedback when they have strong opinions).