JMIR Publications

ABSTRACT

Background:

Measurement-Based Care (MBC) improves patient outcomes by utilizing standardized scales, but its widespread adoption is hindered by the lack of accessible and structured knowledge, particularly in unstructured Chinese medical literature. Extracting scale-related knowledge entities from these texts is challenging due to limited annotated data. While Large Language Models (LLMs) show promise in Named Entity Recognition (NER), specialized prompting strategies are needed to accurately recognize medical scale-related entities, especially in low-resource settings.

Objective:

This study aims to develop and evaluate MedScaleNER, a task-oriented prompt framework designed to optimize LLM performance in recognizing medical scale-related entities from Chinese medical literature.

Methods:

MedScaleNER incorporates demonstration retrieval within in-context learning, chain-of-thought prompting, and self-verification strategies to improve performance. The framework dynamically retrieves optimal examples using a k-Nearest Neighbors approach and decomposes the NER task into two subtasks: entity type identification and entity labeling. Self-verification ensures the reliability of the final output. A dataset of manually annotated Chinese medical journal articles was constructed, focusing on three key entity types: scale names, measurement concepts, and measurement items. Experiments were conducted by varying the number of examples and the proportion of training data to evaluate performance in low-resource settings. Additionally, MedScaleNER’s performance was compared with locally fine-tuned models.

Results:

The CMedS-NER dataset, containing 720 articles with 27,499 manually annotated scale-related knowledge entities, was used for evaluation. Initial experiments identified GLM-4-0520 as the best-performing LLM among six tested models. When applied with GLM-4-0520, MedScaleNER significantly improved NER performance for scale-related entities, achieving a macro-F1 score of 59.64% in exact string match with the full training dataset. In low-resource settings, MedScaleNER outperformed traditional models like BiLSTM-CRF (Chinese-BERT-wwm) and remained competitive with W2NER (MacBERT) when using only 1% to 5% of the training data. The highest performance was achieved with 20-shot demonstrations. Ablation studies highlighted the importance of self-verification in improving model reliability. Error analysis revealed four main types of mistakes: identification errors, type errors, boundary errors, and missing entities, indicating areas for further improvement.

Conclusions:

MedScaleNER advances the application of LLMs and prompt engineering for specialized NER tasks in Chinese medical literature. By addressing the challenges of unstructured texts and limited annotated data, MedScaleNER facilitates more efficient and reliable knowledge extraction, contributing to broader MBC implementation and improved clinical and research outcomes.