JMIR Publications

ABSTRACT

Background:

Intelligent diagnosis and treatment technology in the field of TCM is often based on knowledge graphs, deep learning visual models and machine learning approaches. Traditional supervised learning is limited by the quality and size of the training data, as well as the difficulty of fusion with natural language and its limited generalisability.

Objective:

Develop and validate a vertical model of the traditional Chinese medicine (TCM) domain with TCM understanding and reasoning capability for tongue images.

Methods:

A TongueVLM multi-modal large model is designed, which includes a visual encoder module, a modal fusion module, and a language decoder module. First, the visual encoder based on the CLIP-ViT pretrained model is used for image patch, dimensionality reduction, and migration learning, which maps the high-dimensional tongue features into low-dimensional language encoding vectors. Further, a modal fusion module with a residual architecture is applied to map visual features to natural language word embedding space, realising the conceptual alignment between visual encoding and TCM terminology. Finally, fine-tuning of visual instructions is performed based on the LLaMA, and a TCM-domain LLM with 7B parameters is trained.

Results:

The experimental results show that the TongueVLM model outperforms the general-purpose large model on all three tasks. It is 9.1%, 8.4% and 1.1% more accurate than LLaVA-OneVision and 7.5%, 7.0% and 5.9% more accurate than Qwen2.5-VL-7B.

Conclusions:

The TongueVLM model that achieves tongue image description generation and constitution reasoning in TCM and is suitable for the application of a Chinese medicine intelligent diagnosis system.