JMIR Publications

ABSTRACT

Background:

An effective and scalable information retrieval (IR) system can help clinicians and researchers utilize free-text information in electronic health records. In our previous study, a prototype medical IR system, with a semantically-based query recommendation (SBQR) feature was developed and empirically evaluated, demonstrating high perceived system performance by end users. This follow-up study explored potential factors contributing to the perceived performance through query log analysis.

Objective:

A common challenge of IR is that users may have limited knowledge of what they are searching for. An IR system needs to be carefully designed, especially its user interface, to help users work through this “berry-picking” process where user information needs and queries change when relevant documents are found in the search process. Our solution to addressing these issues is to use “query recommendation”, which has been an integral component in modern IR systems such as Google and Microsoft Bing.

Methods:

The query log data analyzed in the present paper was collected from our previous experimental study where an electronic medical record search engine (EMERSE) with the SBQR feature was developed. A logging mechanism was developed to keep track of the user query behaviors and the IR system output (retrieved documents). In this study, the initial query that a user entered was compared with the query achieved with the assistance of the SBQR. The results of this comparison can determine whether the use of the SBQR helped in constructing improved queries that were different from the original query.

Results:

The results show that the first SBQR-off query and last SBQR-on query were 77% similar, suggesting that the perceived positive performance of the system was more likely due to the automatic query expansion by the SBQR, rather than the manual manipulation of the queries by the users. Moreover, the Entropy analysis showed that search results converged in the scenarios with medium difficulty, and the level of convergence was highly correlated with the perceived performance.

Conclusions:

These findings suggest that the SBQR can contribute to participants’ positive perceptions of performance, contingent upon the scenario difficulty. Medical IR systems should consider designing an SBQR as a user-controlled option or semi-automated feature. Future work involves redesigning the experiment in a more controlled manner and conduct multi-site studies to demonstrate the effectiveness of EMERSE with SBQR for patient cohort identification.