Accepted for/Published in: JMIR Medical Informatics
Date Submitted: May 18, 2022
Date Accepted: Sep 7, 2022
Evaluating the impact on clinical task efficiency of a natural language processing algorithm for searching medical documents: Prospective crossover study
ABSTRACT
Background:
Information retrieval (IR) from the free text within Electronic Health Records (EHRs) is time-consuming and complex. We hypothesise that Natural Language Processing (NLP)-enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians.
Objective:
To evaluate the efficacy of three levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment.
Methods:
A clinical environment was simulated by uploading three sets of patient notes into an EHR research software application and presenting these alongside three corresponding IR tasks. Tasks contained a mixture of multiple choice and free text questions. A prospective crossover study design was used, for which three groups of evaluators were recruited, comprised of doctors (n=19) and medical students (n=16). Evaluators performed the three tasks using each of the search functionalities in an order according to their randomly assigned group. The speed and accuracy of task completion was measured and analysed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey.
Results:
NLP-enhanced search facilitated significantly more accurate task completion than both string search (5.26%, p=0.01) and no search (7.44%, p=0.05). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed over no search function (15.9%/11.6%, p=0.05). 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load.
Conclusions:
To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.