JMIR Publications

ABSTRACT

Background:

Test-time scaling has emerged as a promising method to enhance the reasoning capabilities of large language models (LLMs) and vision-language models (VLMs) during inference without additional training. While interest in medical AI is growing, the effectiveness of these strategies for VLMs and the optimal configurations for varying medical contexts remain underexplored.

Objective:

This study aims to conduct a comprehensive investigation of test-time scaling in the medical domain to identify model-aware and task-aware strategies. The study evaluates the impact of scaling on both LLMs and VLMs across different model sizes and task complexities, and assesses the robustness of these strategies against user-driven factors such as misleading information.

Methods:

The study evaluated a diverse set of models, including general and medical-specific LLMs and VLMs. Experiments utilized five textual medical benchmarks comprising over 5,500 questions, and two multimodal benchmarks comprising 7,000 samples. Performance was measured using accuracy and coverage under three scaling conditions: increasing token budgets (up to 8,192 tokens), iterative sequential scaling, and parallel scaling. Robustness was tested by embedding misleading hints (varying by tone and expertise) into prompts.

Results:

For non-reasoning LLMs, accuracy saturated quickly with token usage often remaining under 500 tokens regardless of budget increases. In contrast, reasoning models demonstrated significant performance gains on complex tasks as token budgets increased. Regarding scaling strategies, parallel scaling outperformed sequential scaling on easier tasks, where additional sequential steps led to accuracy declines; conversely, sequential scaling or increased token budgets proved more effective for complex reasoning and calculation tasks. For VLMs, test-time scaling showed limited effectiveness, with only the QVQ model exhibiting consistent accuracy improvements on the challenging dataset. Finally, while optimal scaling strategies improved robustness against misleading "expert" prompts, they did not fully restore performance to baseline levels.

Conclusions:

Longer reasoning is not universally beneficial in the medical domain; concise reasoning with parallel scaling is optimal for simpler tasks, while extended chain-of-thought via sequential scaling or increased budgets is required for complex problem-solving. Current medical VLMs and non-reasoning LLMs show limited benefit from simple token budget expansion. Effective implementation of test-time scaling requires strategies tailored to specific model characteristics and task difficulty levels to ensure reliability and robustness.