JMIR Publications

ABSTRACT

Background:

Pathology reports contain key information about the patient’s diagnosis as well as important gross and microscopic findings. These information-rich clinical reports offer an invaluable resource for clinical studies, but data extraction and analysis from such unstructured texts is often manual and tedious. Thus, an automated data extraction method from pathology reports would be of significant value and utility.

Objective:

to develop a language model for renal transplant-pathology reports to extract the answers for two predefined questions.

Methods:

The study aimed to automatically answer two pre-defined questions given text from renal pathology reports: 1) “What kind of rejection does the patient show?”; and 2) “What is the grade of interstitial fibrosis and tubular atrophy (IFTA)?”. First, we followed the conventionally recommended procedure and pre-trained Clinical BERT further with the corpus which contains 3.4K renal transplant pathology reports and 1.5M words using Masked Language Modeling to obtain ‘Kidney BERT’. Second, we hypothesized that the conventional pre-training procedure fails to capture the intricate vocabulary of narrow technical domains. We created extended Kidney BERT (‘exKidneyBERT’) by extending the tokenizer of Clinical BERT with six technical keywords from our corpus (which we determined were missing from the original tokenizer vocabulary) and then repeating the pre-training procedure. Third, to further improve performance, all three models were fine-tuned with question-answering (QA) heads tailored to the two questions of interest.

Results:

For the first question regarding rejection, the overlap ratio at word level for exKidneyBERT – 83.3% for antibody-mediated rejection (ABMR) and 79.2% for T-cell mediated rejection (TCMR) – beats that of both Clinical BERT and Kidney BERT (both are 46.1% for ABMR, and 65.2% for TCMR). For the second question regarding IFTA, the exact match rate of exKidneyBERT (95.8%) beats that of Kidney BERT (95.0%) and Clinical BERT (94.7%),

Conclusions:

We developed exKidneyBERT, a high-performing model for automatically extracting information from renal pathology reports. More broadly, we found that when working in domains with highly specialized vocabulary, it is essential to extend the vocabulary library of the BERT tokenizer to improve model performance, otherwise, pre-training (especially on small corpuses) is ineffective. In our case, pre-training BERT language models for kidney pathology reports improved model performance even though the training data were relatively small.