JMIR Publications

ABSTRACT

Background:

Named Entity Recognition (NER) models play a pivotal role in deciphering unstructured medical texts by identifying diseases, treatments, and conditions, thereby advancing clinical decision-making and research. Machine learning innovations, especially in deep learning, have notably enhanced NER capabilities. Yet, their performance is inconsistent across medical datasets due to the complexity of medical terminology and linguistic variety. Prior studies have predominantly analyzed general NER performance, overlooking specific applications in medical scenarios and the challenges therein. Moreover, an in-depth analysis of how leading models and macro-factors, such as linguistic composition, affect NER accuracy is needed. This deficiency impedes the refinement of NER models for medical applications, which is vital for improving patient outcomes and the efficiency of healthcare services.

Objective:

This study aims to meticulously evaluate the performance of BioBERT, RoBERTa, BigBird, and DeBERTa NER models within medical text analysis, concentrating on varied medical datasets to determine how complex medical terminology and linguistic diversity affect entity recognition accuracy. It also examines the role of macro-factors, including the lexical composition of entity phrases, in influencing the efficacy of specific models. The goal is to bridge the current research gap by offering insights that facilitate refining NER models for medical use, ultimately advancing patient care and healthcare service efficiency.

Methods:

This study conducts a thorough evaluation of four prominent NER models: BioBERT, RoBERTa, BigBird, and DeBERTa. The focus is assessing prediction accuracy, training efficiency, computational resource use (CPU and GPU), etc. We utilized three diverse medical datasets-Revised JNLPBA, BC5CDR, and AnatEM-selected for their relevance to the medical field. Furthermore, the study explores the impact of significant macro-factors, like the number of words in an entity phrase, on the models’ performance. A systematic analysis of these factors’ influence on prediction accuracy across the datasets was performed, aiming to gain an in-depth understanding of the impact of different macro-factors on the prediction accuracy of the medical NER model.

Results:

The analysis shows that the BioBERT model exceeded the performance of other models in prediction accuracy across the Revised JNLPBA, BC5CDR, and AnatEM medical datasets, highlighting its superior proficiency in identifying medical entities. Nevertheless, its accuracy was not consistently superior across all entity types. Additionally, the research confirmed that macro-factors, such as the number of words in an entity phrase, markedly affect the prediction accuracy of the models.

Conclusions:

This study highlights the essential role of NER models in medical informatics, emphasizing the imperative for model optimization via precise data targeting and fine-tuning. The insights from this study will notably improve clinical decision-making and facilitate the creation of more sophisticated and effective medical NER models.