JMIR Publications

ABSTRACT

Background:

Clinical Natural Language Processing (NLP) researchers need access to directly comparable evaluation results for applications such as text de-identification across a range of corpus types and the means to easily test new systems or corpora within the same framework. Current systems, reported metrics, and personally identifiable information (PII) categories evaluated are not easily comparable.

Objective:

This study presents an open-source and extensible end-to-end evaluation framework for comparing clinical NLP system performance across corpora even when the annotation categories do not align.

Methods:

As use case for this new evaluation framework, we use 6 off-the-shelf text de-identification systems (CliniDeID, deid from Physionet, MIST, NeuroNER, NLM Scrubber, and Philter) across three standard clinical text corpora for the task (two of which are publicly available) and one private corpus (all in English) for a total of four corpora, with annotation categories that are not directly analogous. The framework is built on shell scripts that can be extended to include new systems, corpora, and performance metrics. We present this open tool, multiple means for aligning PII categories during evaluation, and our initial timing and performance metric findings. Code for running this framework with all configuration file settings needed to run all pairs are available via Codeberg (https://codeberg.org/HeiderLab/article-addenda) and GitHub (https://github.com/MUSC-TBIC/article-addenda).

Results:

From this case study, we found an order of magnitude difference in processing speed between systems. The fastest system (MIST) processed an average of 24.57 notes/second while the slowest system (CliniDeID) processed an average of 1.00. No single system uniformly outperformed the others at identifying PII across corpora and categories. Instead, a rich tapestry of performance trade-offs for PII categories and groups of categories appeared. CliniDeID and Philter prioritize recall over precision (with an average recall 6.9 and 11.2 points higher, respectively, for partially matching spans of text matching any PII category) while the other four systems consistently have higher precision (with MIST's precision 20.2 points higher, NLM Scrubber +4.4 points, NeuroNER +7.2 points, and deid +17.1 points). The macro-average recall across corpora for identifying names, one of the more sensitive PII categories, included deid (48.8%) and MIST (66.9%) at the low end and NeuroNER (84.1%), NLM Scrubber (88.1%), and CliniDeID (95.9%) at the high end. A variety of metrics across categories and corpora are reported with an even wider variety (e.g., F2-score) available via the tool.

Conclusions:

NLP systems in general, and de-identification systems (and new domain corpora) in our use case, tend to be evaluated in stand-alone research articles that only include a limited set of comparator systems. We hold that a single evaluation pipeline across multiple systems and corpora allows for more nuanced comparisons. The open pipeline we present should reduce barriers to evaluation and system advancement.