JMIR Publications

ABSTRACT

Background:

General practitioners are confronted with a wide variety of diseases and sometimes diagnostic uncertainty. Clinical decision support systems could be valuable to improve diagnosis, especially for unusual manifestations of diseases and rare diseases. Existing tools are not adapted to the requirements and workflow in the primary setting. In the project SATURN (Smart physician portal for patients with unclear disease), the prototype of a clinical decision support system based on artificial intelligence is being developed together with users specifically for primary care in Germany. It focuses on three medical fields. A user-centered design approach is applied for prototype development and evaluation.

Objective:

This study evaluates the usability of a high-fidelity prototype and explores aspects of user experience like the subjective impression, satisfaction and areas of improvement.

Methods:

Five general practitioners participated in the evaluation of the prototype which consisted of (1) a remote think-aloud test, (2) a post-session interview, and (3) a survey with the System Usability Scale. All three parts were consecutively carried out in individual remote sessions. During the think-aloud tests, which were video- and audiotaped, the participants verbalized their thoughts and actions and had to solve several tasks which were based on a primary care case vignette. Remarkable observations were logged, transcribed with quotes, and analyzed for usability problems and positive findings. All observations and interview responses were deductively assigned to the following categories: (1) Content, (2) Comprehensibility, (3) User-friendliness, (4) Layout, (5) Feedback, (6) Navigation. Usability problems were described in detail and solutions for improvement proposed. Median and total scores were calculated for all questionnaire items.

Results:

The evaluation detected both strengths and areas for improvement. Key issues identified were content-related limitations, such as the inability to enter unlisted symptoms, medications, and examination findings in the dropdown menus. Participants also found the terminology for laboratory values did not match their day-to-day vocabulary, as common abbreviations were not recognized. Suggestions for improving the content of the system were also made and included adding symptom duration, weighting symptoms, and incorporating hereditary factors. Another key issue was a lack of user-friendliness concerning the time required to input medication plans and lab values. This aspect was criticized for being cumbersome, with participants expressing a need for faster data entry, potentially through direct imports from practice management systems or laboratory files. Despite these challenges, participants praised other aspects of user-friendliness (use of stored diagnoses and symptoms) and navigation (top navigation bar), and particularly liked the clear and well-structured layout. Overall, the SATURN prototype was deemed useful and promising for future clinical use, despite the need for further refinements, particularly in the areas of data entry, as this is a key obstacle to its use.

Conclusions:

The usability evaluation methods combined proved to be location independent and easy to use, and were apt to detect usability problems in detail. They provided important findings on usability issues and improvements that will be implemented in a second high-fidelity prototype, which will also be tested by users. Technically demanding user requirements, such as direct data transfer from the practice management system and entry options that require complex data models were beyond the scope of this project. However, they should be considered in future CDSS-development projects.