JMIR Publications

ABSTRACT

Background:

Social media can provide valuable information for public health surveillance, but detecting valid symptom expressions in noisy and informal text remains challenging. While fine-tuned language models can achieve strong test-set accuracy, they are prone to relying on spurious patterns rather than surveillance-relevant features, which limits their reliability in real-world applications.

Objective:

This study aims to develop a more robust symptom detection model by explicitly incorporating surveillance-relevant attributes into the training objective and by introducing a counterfactual-based evaluation framework to better assess robustness.

Methods:

We propose the Concept-Aware Learning Framework (CALF). This strategy allows one to identify a set of surveillance-relevant concepts, and incorporate it in the fine-tuning process. CALF does this by integrating multi-attribute counterfactual data augmentation with a multi-task learning objective. Different model fine-tuning strategies were compared on a multi-label symptom classification dataset, and model robustness was further probed using a newly designed counterfactual-based test.

Results:

All training strategies achieved comparable macro F1 scores on the standard test set (range: 0.84-0.89). However, substantial differences emerged under the counterfactual-based test. Based on the counterfactual-based test, CALF achieved the highest overall flip accuracy (0.82; 95% CI: 0.80–0.84), significantly outperforming standard supervised fine-tuning (0.45; 95% CI: 0.43–0.47), data-augmentation-only approaches, and alternative hybrid methods. While the baseline model was primarily sensitive to direct symptom mentions, CALF substantially improved sensitivity to other surveillance-relevant concepts, including recovery status, timing of occurrence, and uncertainty. Robustness gains were also consistent across most symptom categories and pretrained models, and qualitative error analysis showed that CALF was less likely to misclassify historical, speculative, or informational mentions as positive cases.

Conclusions:

High test-set accuracy is not a good indicator of model robustness. Standard supervised fine-tuning performed well on the test set, but the counterfactual-based test implies that it is not robust to data-specific biases. In contrast, CALF maintained good performances in both the test set and the counterfactual-based test. This indicates CALF allows one to train a model that is both accurate and sensitive to surveillance-relevant concepts.