JMIR Publications

ABSTRACT

Background:

In response to the intricate language and specialized terminology inherent in healthcare text data, domain adaptation techniques have emerged as crucial to Transformer-based models for improved performance in downstream tasks, such as Information Extraction.

Objective:

This study presents a systematic literature review investigating domain adaptation methods for Transformers in healthcare differentiating between English and non-English languages, with a focus on Portuguese, addressing respective language-specific challenges, and information extraction models to assess the efficacy of Transformer-based approaches in healthcare contexts.

Methods:

This systematic review was conducted using the Preferred Reporting Items for Systematic Reviews and Meta-analyzes (PRISMA) methodology on Scopus and Web of Science Core Collection databases. Only studies that mentioned the creation of healthcare language models or healthcare information extraction models were included, while large language models were excluded.

Results:

Our search query retrieved 137 articles, 60 of which met the inclusion criteria. Furthermore, 5 studies were included via other methods. For studies mentioning the creation of healthcare language models, 7 were trained using English data, 4 using Chinese data, and only 3 using Portuguese, with 2 of them being Brazilian Portuguese. Regarding Information Extraction models, Transformers were the most commonly utilized method, with 16 articles, while other Deep Learning or Machine Learning methods were also popular. Nevertheless, the current state-of-the-art for Information Extraction models involves utilizing Transformers. The most extracted entities were diagnosis, posology, symptoms, and phenotype related to specific diseases.

Conclusions:

The findings indicate that domain adaptation is beneficial for the healthcare context, achieving better results in downstream tasks. Our analysis allowed us to understand that the utilization of Transformers is more developed for the English language, while non-English languages face challenges regarding the lack of available data, resources, or relevant studies, with the Chinese language having the most results. The Portuguese language lacks relevant studies in its European version and should draw examples from other non-English languages to develop these models and drive progress in AI. For the Information Extraction task, several methods were employed, but the current state-of-the-art utilizes Transformers.