JMIR Publications

ABSTRACT

Background:

Small Language Models (SLMs) running on local hardware can reduce the privacy, latency, and cost concerns associated with cloud-based Large Language Models (LLMs) in healthcare, but deciding when to use them is typically informal and unsystematic in current practice. The problem is compounded by model turnover: benchmark results tied to specific models begin to date as soon as they are published.

Objective:

This paper introduces FIT-SLM-HC (Framework for Identifying Tasks for Small Language Models in Healthcare), a framework rooted in Task–Technology Fit theory for structuring and prioritizing the evaluation of clinical tasks as candidates for on-device SLMs. The framework is deliberately model-agnostic: it favors no particular model, but specifies an initial framework that can be re-run as models, hardware, and practices evolve.

Methods:

The framework separates task-side properties, scored along axes of Reasoning Complexity, Knowledge Boundedness, and Output Structure, from system-side metrics: Accuracy Ratio (AR) and Latency Efficiency (LE), computed relative to a named reference model with an absolute latency target. An SLM capability envelope is the set of tasks for which a specific system meets risk-adjusted thresholds. The paper separates the parts of the framework that should outlast today’s models from the parts that will not: (1) the scoring and measurement procedure is model-agnostic, (2) the prediction that smaller models fall further behind as tasks demand more stored knowledge and deeper reasoning is likely to hold across most general model generations, and (3) the accompanying snapshot of published 2024–2026 healthcare benchmarks (22 unique tasks, 11 studies) is time-stamped and expected to date.

Results:

The primary result is the framework itself, composed of three task axes, the two system metrics, and a decision procedure that they define. Three vignettes illustrate the framework: (1) a pathology extraction task lands inside the envelope (AR = 0.94, LE ≈ 0.50– 0.58), (2) a clinical note summarization task falls outside on content recall (ROUGE-1 AR = 0.73), and (3) a wearable fatigue-prediction task where AR flips from AR = 1.53 to AR = 0.87 depending on the reference model applied. Across the larger historical snapshot, 16 of the 18 unique tasks with axis sum ≤ 5 reached AR ≥ 0.90 and 13 exceeded 1.0, most under SLM-favoring adaptation asymmetry. Of the three axes, only reasoning complexity shows a level-by-level decline in mean AR (1.12, 1.09, 0.88) with exam-related tasks. Importantly the output-structure axis was untested due to the available literature.

Conclusions:

FIT-SLM-HC acts as a first-pass triage tool that gives healthcare organizations a structured procedure for helping determine when to default to local SLMs versus cloud-based LLMs via explicit metric selection, reference-model selection, and threshold selection. The framework is designed to be re-run as models change rather than once, thus rapid model turnover due to improvements strengthens rather than weakens the case for this approach. Blinded multi-annotator validation of the task rubric and prospective evaluation are necessary but not yet performed. Clinical Trial: NA