JMIR Publications

ABSTRACT

Background:

Large Language Models (LLMs) are increasingly integrated into healthcare, where they contribute to patient care, administrative efficiency, and clinical decision-making. Despite their growing role, the ability of LLMs to handle imperfect inputs remains underexplored. These imperfections, which are common in clinical documentation and patient-generated data, may affect model reliability.

Objective:

This study investigates the impact of input perturbations on LLM performance across three dimensions: (1) overall effectiveness in different health-related applications, (2) comparative effects of different types and levels of perturbations, and (3) differential impact of perturbations on health-related terms versus non-health-related terms.

Methods:

We systematically evaluate three LLMs on three health-related tasks using a novel dataset containing three types of human-like variations (redaction, homophones, and typographical errors) at different perturbation levels.

Results:

Contrary to expectations, LLMs demonstrate notable robustness to common variations, with some cases showing improved performance at lower perturbation levels. Redactions, often stemming from privacy concerns or cognitive lapses, are more detrimental than other variations.

Conclusions:

Our findings highlight the need for healthcare applications powered by LLMs to be designed with input variability in mind. Robustness to noisy or imperfect inputs is essential for maintaining reliability in real-world clinical settings, where data quality can vary widely. By identifying specific vulnerabilities and strengths, this work provides actionable insights for improving model resilience and guiding the development of safer, more effective AI tools in healthcare. The accompanying dataset offers a valuable resource for further research into LLM performance under diverse conditions. Clinical Trial: (N/A)