JMIR Publications

ABSTRACT

Background:

Event-based surveillance (EBS) relies on large volumes of open-source reports to support the early detection of infectious diseases; however, the information far exceeds the capacity for manual review. Many existing approaches focus on entity extraction, while extraction for disease mentions alone does not necessarily indicate active infectious disease events. They also face challenges related to semantic interpretation, non-standard and evolving terminology, and the linguistic diversity of open-source reports, limiting their ability to distinguish true infectious disease events from non-event information.

Objective:

This study aimed to develop a rule- and LLM-based automated framework incorporating predefined epidemiological criteria to identify infectious disease events and extract key epidemiological attributes from open-source surveillance reports.

Methods:

We collected 7,000 reports between January 2023 and December 2025 from four EBS systems: Epidemic Intelligence from Open Sources (EIOS), Program for Monitoring Emerging Diseases (ProMED), ReliefWeb, and WHO Disease Outbreak News (WHO DONs). Reports were selected to obtain enough diversity from each data source for sampling. The framework integrated predefined fixed rules with structured prompts guiding DeepSeek-3.1 model, with each approach applied according to the characteristics of the specific task. The analytical workflow proceeded through sequential stages to determine whether the reports describe an infectious disease event, extract and normalize disease names and locations, and classify affected categories. Framework outputs were compared against manual annotations by public health researchers as the standard. Performance was evaluated using sensitivity, positive predictive value (PPV), and extraction accuracy for disease names and countries.

Results:

Among the 7,000 sampled reports, 1,580 (22.6%) were manually confirmed as true infectious disease events. The framework achieved a sensitivity of 93.7% and a PPV of 95.2% for event identification. Among correctly identified events, disease name extraction accuracy was 98.0% and country extraction accuracy was 98.1%. Disease extraction errors resulted from overgeneralization to broad disease categories, whereas country extraction errors were mainly associated with overseas territories.

Conclusions:

This framework demonstrate high performance on identifying infectious disease events and extracting key epidemiological attributes from open-source surveillance reports. By integrating predefined rules with LLM-based semantic interpretation, the framework can effectively distinguish surveillance-relevant events from large volumes of non-event information, which may support epidemic intelligence activities in routine public health surveillance.