JMIR Publications

ABSTRACT

Background:

Named Entity annotation is a fundamental task in Natural Language Processing. However, it has associated challenges. Especially in the clinical domain, determining entity boundaries is one of the most common sources of disagreements between annotators, as questions such as whether modifiers or peripheral words should be annotated appear. If unresolved, these can induce inconsistency in the produced corpora, but, on the other hand, strict guidelines or adjudication sessions prolong what is already a slow and convoluted process.

Objective:

To address these challenges, we evaluate two novel annotation methodologies: Lenient Span and Point annotation, aiming to relieve the pain of precisely determining entity boundaries.

Methods:

We evaluate their effects through an annotation case study on a dataset of Japanese medical case reports. We compare annotation time, annotator agreement, and the quality of the produced labeling and assess the impact on the performance of a NER system trained on the annotated corpus.

Results:

We saw significant improvements in the labeling process efficiency, with up to a 25% reduction in overall annotation time and even an 8% improvement in annotator agreement compared to the traditional boundary-strict approach. However, even the best-achieved NER model presented some drop in performance.

Conclusions:

Our findings demonstrate a balance between annotation speed and model performance. Although disregarding boundary information affects model performance to some extent, this is counterbalanced by significant reductions in the annotator’s workload and notable improvements in the speed of the annotation process. These benefits may prove valuable in various applications, offering an attractive compromise for developers and researchers.