JMIR Publications

ABSTRACT

Background:

Several clinical cases and experiments have demonstrated the effectiveness of traditional Chinese medicine (TCM) formulas in treating and preventing diseases. These formulas encapsulate critical information regarding their ingredients, efficacy, and indications. Classifying TCM formulas based on this information can effectively standardize their management, support clinical and research application, and promote the modernization and scientific use of TCM. To further advance this task, TCM formulas can be classification using various approaches, including manual classification, machine learning, and deep learning. Additionally, large language models (LLMs) are gaining prominence in the biomedical field. Integrating LLMs into TCM research could significantly enhance and accelerate the discovery of TCM knowledge by leveraging their advanced linguistic comprehension and contextual awareness.

Objective:

The objective of this study is to assess the accuracy of different LLMs in TCM formulas classification task. Additionally, by utilizing ensemble learning, our study aims to improve the accuracy of the task by utilizing multiple fine-tuned LLMs.

Methods:

The data for the TCM formula was manually refined and cleaned. We selected ten LLMs that support Chinese for fine-tuning. We then used an ensemble learning approach to subject the model results to both hard and weighted voting, with weights determined by the average accuracy of each model. Finally, we selected the top five most effective models from each series of LLMs for weighted voting (Top5), and the top three most accurate models out of ten for weighted voting (Top3).

Results:

A total of 2,441 formulas were curated manually from various sources, including the Coding Rules for Chinese Medicinal Formulas and Their Codes, the Chinese National Medical Insurance Catalog for proprietary Chinese medicines, textbooks of formulas of Chinese medicine, and TCM literature. The training and testing sets consisted of 1,999 and 442 TCM formulas, respectively. The testing results showed that Qwen-14B achieved the highest accuracy of 75.32% among the single models. The accuracy rates for hard voting, weighted voting, weighted voting (Top 5), and weighted voting (Top 3) were 75.79%, 76.71%, 75.57%, and 77.15%, respectively.

Conclusions:

The primary objective of this study is to explore the effectiveness of LLMs in the classification task. To achieve this, an ensemble learning method is proposed that integrates multiple fine-tuned LLMs through a voting mechanism. This approach not only improves accuracy but also enables improvement on the existing classification system of TCM formula efficacy.