JMIR Publications

ABSTRACT

Background:

Online medical information in colon surgery, particularly in robotic-assisted procedures, is frequently inconsistent, outdated, or misleading, which can hinder patient education and informed decision-making. Large Language Models (LLMs) have shown promise in complex text analysis, but their potential for assessing the quality of medical web content has not been systematically explored. The Ensuring Quality Information for Patients (EQIP) 36 criteria offer a standardized framework for evaluating patient-facing materials, but manual assessments are time-consuming and resource-intensive.

Objective:

To systematically evaluate the ability of multiple transformer-based LLMs to classify the quality of online information on robotic colon surgery according to the EQIP 36 criteria, and to determine whether embedding-oriented or generative models provide better performance for this task.

Methods:

English-language web pages on robotic colon surgery were collected in December 2023 via automated searches on Google, Bing, and Yahoo using predefined keywords. From 450 retrieved URLs, duplicates, irrelevant content, and professional/academic sites were removed, resulting in 47 unique patient-targeted websites with expert EQIP ratings. Due to input length limitations, texts were split into overlapping 512-token chunks (50-token overlap) before embedding extraction. Models evaluated included base BERT, custom pre-trained BERT, BiomedBERT, pre-trained BiomedBERT, SBERT, LLaMA-2 (concatenated and averaged embeddings), and Longformer. Chunk embeddings were concatenated into document-level embeddings and classified via a feedforward multilabel neural network predicting 19 EQIP items. Hyperparameters (batch size, learning rate, dropout) were optimized via grid search. Performance was assessed using stratified 5-fold cross-validation, with accuracy, precision, recall, F1-score, and ROC-AUC as metrics.

Results:

SBERT achieved the highest performance (F1-score = 80.16, accuracy = 79.05%, AUC = 0.79), outperforming all other models. Pre-trained BiomedBERT and custom pre-trained BERT performed comparably well (F1 ≈ 0.79), indicating benefits from domain-specific adaptation. LLaMA-2 with concatenated embeddings (F1 = 0.784) outperformed the averaged embedding version (F1 = 0.765) but remained below SBERT. Longformer, despite handling longer contexts, had the lowest performance (F1 = 70.07). Across models, those optimized for generating semantically meaningful embeddings consistently surpassed generative architectures.

Conclusions:

Transformer-based models optimized for embedding generation, particularly SBERT, are more effective than large generative models in automated EQIP-based quality assessment of online medical content. Domain-specific adaptation improves performance, but embedding quality is a more critical determinant than model size. These findings support the integration of embedding-focused LLMs into scalable, AI-driven tools for medical content evaluation, potentially enhancing the accessibility and reliability of online patient education. Future research should expand datasets, explore ensemble approaches, and address regulatory and ethical considerations for deployment in healthcare contexts.