JMIR Publications

ABSTRACT

Background:

The volume of digital data in healthcare is continually growing. In addition to being used in healthcare, the health data collected can also be used for secondary purposes, such as research. In this context, Clinical Data Warehouses (CDW) provide the infrastructure and organization needed to improve the secondary use of health data. Various data models have been proposed for organizing data in a CDW, including the i2b2 model, whose persistence is based on a relational database that can present performance problems when executing queries on massive data.

Objective:

The objective of this work is to describe the required transformation and the implementation of data persistence for part of the i2b2 model in a NoSQL Elasticsearch database.

Methods:

A comparison is made with data persistence in a standard relational database, in terms of query response, execution performance and in terms of material resource requirements. A description of the data loading and updating processes is also provided.

Results:

We propose adaptations of the i2b2 model to take into account the specific features of Elasticsearch, in particular the impossibility of performing joins between different indexes. The implementation has been tested and evaluated within the CDW of the Bordeaux University Hospital, which includes data on 2.5 million patients and more than 3 billion observations. Overall, elasticsearch query execution times are shorter than with a relational database. The performance gain is particularly significant for queries involving free-text searches. Compared with an indexed relational database (including a full-text index), the disk space required for storage is smaller with elasticsearch.

Conclusions:

We demonstrate that an Elasticsearch implementation is feasible, with a significant improvement in query performance and for disk space used for storage. This implementation is currently used in production at Bordeaux University Hospital.