JMIR Publications

ABSTRACT

Background:

As a risk factor for many diseases, family history captures both shared genetic variations and living environments among family members. Though there are several systems focusing on family history extraction (FHE) using natural language processing (NLP) techniques, the evaluation protocol of such systems has not been standardized.

Objective:

The n2c2/OHNLP 2019 FHE Task aims to encourage the community efforts on a standard evaluation and system development on FHE from synthetic clinical narratives.

Methods:

We organized the first BioCreative/OHNLP FHE shared task in 2018. We continued the shared task in 2019 in collaboration with n2c2 and OHNLP consortium, and organized the 2019 n2c2/OHNLP FHE track. The shared task composes of two subtasks. Subtask 1 focuses on identifying family member entities and clinical observations (diseases), and Subtask 2 expects the association of the living status, side of the family and clinical observations to family members to be extracted. Subtask 2 is an end-to-end task which is based on the result of Subtask 1. We manually curated the first de-identified clinical narrative from family history sections of clinical notes at Mayo Clinic Rochester, the content of which are highly relevant to patients’ family history.

Results:

17 teams from all over the world have participated in the n2c2/OHNLP FHE shared task, where 38 runs were submitted for subtask 1 and 21 runs were submitted for subtask 2. For subtask 1, the top three systems were generated by Harbin Institute of Technology, ezDI, Inc, and The Medical University of South Carolina with F1 scores of 0.8745, 0.8225, and 0.8130, respectively. For subtask 2, the top three systems were generated by Harbin Institute of Technology, ezDI, Inc, and University of Florida with F1 scores of 0.681, 0.6586, and 0.6544, respectively. The workshop was held in conjunction with the AMIA 2019 Fall Symposium.

Conclusions:

A wide variety of methods were used by different teams in both tasks, such as BERT, CNN, Bi-LSTM, CRF, SVM, and rule-based strategies. System performances show that relation extraction from family history is a more challenging task when compared to entity identification task.