JMIR Publications

ABSTRACT

In the complex and multi-dimensional field of medicine, multimodal data is prevalent and crucial for informed clinical decisions. Multimodal data spans a broad spectrum of data types, including medical images (e.g., MRI, CT scans), time-series data (e.g., sensor data from wearable devices, electronic health records), audio recordings (e.g., heart/respiratory sounds, patient interviews), text (e.g., clinical notes, research articles), videos (e.g., surgical procedures), and omics data (e.g., genomics, proteomics). While advancements in large language models (LLMs) have enabled new applications for knowledge retrieval and processing in the medical field, most LLMs remain limited to processing single-modal data, typically text-based content, and often overlook the importance of integrating the diverse data modalities encountered in clinical practice. This paper aims to present a detailed, practical, and solution-oriented perspective on the use of multimodal large language models (M-LLMs) in the medical field. Our investigation spans M-LLMs foundational principles, current and potential applications, technical and ethical challenges, and future research directions. By connecting these elements, we aim to provide a comprehensive framework that links diverse aspects of M-LLMs, offering a unified vision for their future in healthcare. This approach aims to guide both future research and practical implementations of M-LLMs in healthcare, positioning them as a paradigm shift toward integrated, multimodal data-driven medical practice. We anticipate that this work will spark further discussion and inspire the development of innovative approaches in the next generation of medical M-LLM systems.