Accepted for/Published in: JMIR Human Factors
Date Submitted: May 11, 2021
Date Accepted: Jan 9, 2022
The benefits of involving end-users in the design of formulae for ranking search results
ABSTRACT
Background:
The Internet is a major information source for healthcare professionals; it contributes to improving healthcare quality and lifelong training. However, professionals are drowned in overabundant information; the challenge is how to identify relevant documents. Search engines must sort the search results and present the most relevant documents first.
Objective:
To evaluate two parameter settings of a formula ranking the search results produced by a healthcare literature search engine (LiSSa), using a quantitative and qualitative evaluation method involving the end-users.
Methods:
Ten general practitioners and ten registrars participated in user tests. They first assessed the relevance of the search results and then rated the ranking criteria used in the formulae.
Results:
The method used provided relevant feedback for the selection and improvement of the ranking formula. The formula that prioritized articles representing a consensus in the field was preferred. When users assess an article’s relevance, they primarily judge the topic, the study methods, and its value in clinical practice.
Conclusions:
Our application of a qualitative methodology generated valuable user input on improving LiSSa’s ranking formula and moving towards a highly usable search engine. A user-based evaluation enabled us to design a formula that met the healthcare professionals' needs.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.