Warning: This is an author submission that is not peer-reviewed or edited. Preprints - unless they show as "accepted" - should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information.
SLM outperform LLMs in Classifying Social Determinants of Health in Clinical Records
ABSTRACT
Background:
Social and Behavioral Determinants of Health (SDoH) are non-clinical social and environmental factors – such as income, education, housing, and access to care – that strongly influence health outcomes, are often embedded in unstructured EHR (electronic health record) notes, and remain difficult to systematically identify despite accounting for most modifiable drivers of health disparities.
Objective:
This study contributed expert-derived resources to support the development of high-quality SDoH annotated datasets and compared the performance of smaller transformer-based models and large language models (LLMs) in classifying SDoH snippets.
Methods:
We generated high-quality SDoH snippets using keywords to extract relevant snippets from 50,000 electronic health records, categorized into 20 SDoH types with subcategories (e.g., positive, adverse, uncertain). We integrated annotation schemes from related works and applied four distinct schemes to address real-world needs. Fine-tuning was performed on smaller transformers, including BERT and Gatortron, and state-of-the-art LLMs, such as Llama-3 and Mistral. Model performance was evaluated using macro-averaged F1, Precision, and Recall scores across four annotation levels.
Results:
We curated a high-quality dataset comprising 2,364 annotated SDoH snippets. Among the four annotation levels, Gatortron-S performed best in the first two levels, BioClinical-BERT in the third level, and Llama-3 in the fourth level. Additionally, we released a comprehensive list of expert-derived keywords to facilitate future efforts in SDoH snippet annotation and dataset development.
Conclusions:
We found that smaller transformer-based models demonstrated comparable or superior performance to advanced LLMs when sufficient data were available. This challenges the assumption that LLMs universally outperform smaller models, emphasizes the necessity of high-quality annotated datasets, and provides critical resources and insights for advancing SDoH classification research.
Citation
Request queued. Please wait while the file is being generated. It may take some time.
Copyright
© The authors. All rights reserved. This is a privileged document currently under peer-review/community review (or an accepted/rejected manuscript). Authors have provided JMIR Publications with an exclusive license to publish this preprint on it's website for review and ahead-of-print citation purposes only. While the final peer-reviewed paper may be licensed under a cc-by license on publication, at this stage authors and publisher expressively prohibit redistribution of this draft paper other than for review purposes.