JMIR Publications

ABSTRACT

Background:

In the field of medicine and medical informatics, the importance of comprehensive metadata has long been recognized and the composition of metadata has become its own field of profession and research. To ensure sustainable and meaningful metadata is maintained, standards and guidelines like the FAIR principles have been published. The compilation and maintenance of metadata is done by field experts supported by interactive metadata management applications. The usability of these applications, e.g. in terms of ease of use, efficiency and error tolerance, crucially determines their benefit to those interested in the data.

Objective:

Our goal is to improve the usability of metadata management applications used in collaborative research networks. For this reason, we developed a feasible methodology for remotely performed usability evaluations, including adoptable common tasks and a standardized usability evaluation form. We use this method to measure the usability of the CoMetaR web application and to reveal existing issues.

Methods:

We evaluated the CoMetaR web application by measuring the usability of three modules: “core module”, “provenance module” and “data integration module”. We defined ten tasks in which users must acquire information specific to their user role. The participants were asked to fulfill the tasks in a live web meeting. To measure the application’s usability, we observed the users’ behavior, collected their feedback and applied the System Usability Scale questionnaire.

Results:

A total of twelve subjects participated in the evaluation. We found that over 96 percent of all tasks have been completed successfully. We measured usability scores of 81, 81 and 72 for the three evaluated modules. A qualitative analysis resulted in 24 issues with the application.

Conclusions:

Our newly developed, science-based methodology for usability assessment of metadata repositories can be adapted to evaluate applications within the medical informatics field and potentially beyond. The specific tasks we defined can be reused directly or adjusted for custom evaluations. Further application of our assessment and evaluation methodology will also move the field forwards to ensure high quality tools. For CoMetaR, the usability score of 81 implies very good usability for two modules while a usability score of 72 still indicates acceptable usability for the third module. We identified 24 issues, which will serve as starting points for further development.