JMIR Publications

ABSTRACT

Background:

To meet the growing importance of real-word data analysis, timely availability of clinical data and biosamples is a crucial factor. So-called feasibility platforms are often the first point of contact to determine the availability of such data for specific research questions. Therefore, a user-friendly interface should be provided to make access to this information as easy as possible. The German Medical Informatics Initiative (MII) also aims to establish such a platform for its infrastructure. Although some of these platforms are actively used, their tools usually still have limitations. For this reason, the MII consortium MIRACUM (Medical Informatics in Research and Care in University Medicine) has set itself the task to analyze the pros and cons of existing solutions and to design an optimized graphical feasibility user interface.

Objective:

To identify the system that is most user-friendly and thus forms the best basis for developing a harmonized tool, we carried out a comparative usability evaluation of existing tools conducted by researchers as end users.

Methods:

The evaluation included 3 pre-selected search tools and was conducted as a qualitative exploratory study with a randomized design over a period of 6 weeks. The tools in question were the MIRACUM i2b2 feasibility platform, OHDSI's (Observational Health Data Sciences and Informatics) ATLAS, and the Sample Locator of the German Biobank Alliance. It was carried out in the form of an online usability test (usability walkthrough combined with an online questionnaire) with female and male participants aged 26 to 63 who work as doctors with scientific background.

Results:

In total, 17 study participants evaluated the 3 tools. The overall evaluation of usability based on the System Usability Scale (SUS) showed that the Sample Locator with a mean SUS score of 77.03 (SD=20.62) is significantly superior to the other two tools (Wilcoxon Test; Sample Locator vs. i2b2: P=.047; Sample Locator vs. ATLAS: P=.001). i2b2 with a score of 59.83 (SD=25.36) still performs significantly better than ATLAS with a score of 27.81 (SD=21.79) (Wilcoxon-Test; i2b2 vs. ATLAS: P=.005). The analysis of the material generated by the usability walkthrough method confirmed these findings. Here, ATLAS caused the most usability problems (n=66), followed by i2b2 (n=48) and finally the Sample Locator (n=22). Moreover, the Sample Locator achieved the highest ratings with respect to the additional questions regarding the satisfaction with the tools.

Conclusions:

The present study provides data to develop a suitable basis for the selection of a harmonized tool for feasibility studies by concrete evaluation and comparison of the usability of three different types of query builders. The feedback of the participants during the usability test made it possible to identify user problems and positive design aspects of the individual tools and to compare them qualitatively.