JMIR Publications

ABSTRACT

Background:

In 2021, the European Union reported over 270,000 excess deaths, including more than 16,000 in Portugal. The Portuguese Directorate-General of Health (DGS) developed a deep neural network, AUTOCOD, which determines the primary causes of death by analyzing the free text of physicians' death certificates (DC). Although AUTOCOD's performance has been established, it remained unclear whether its performance remained consistent over time, particularly during excess mortality periods.

Objective:

To assess the sensitivity and other performance metrics of AUTOCOD in classifying underlying causes of death compared to manual coding to identify specific causes of death during excess mortality periods.

Methods:

We included all DCs between 2016 and 2019. AUTOCOD's performance was evaluated by calculating various performance metrics, such as sensitivity, specificity, positive predictive value (PPV), and F1-score, using a confusion matrix. This compared ICD-10 classifications of DCs by AUTOCOD to those by human coders at DGS (gold standard). Subsequently, we compared periods without excess mortality to periods of excess, severe, and extreme excess mortality. Excess mortality was defined as two consecutive days with a Z-score above the 95% baseline limit, severe excess mortality with a Z-score above +4 standard deviations (SD), and extreme excess mortality with a Z-score above +6 SD. Finally, we repeated the analyses for the three most common ICD-10 chapters, focusing on block-level classification.

Results:

We analyzed a large dataset comprising 330,098 death certificates classified by both human coders and AUTOCOD. AUTOCOD demonstrated high sensitivity (≥0.75) for ten ICD-10 chapters examined, with values surpassing 0.90 for the more prevalent chapters (II – neoplasms; IX – diseases of the circulatory system; X – diseases of the respiratory system), accounting for 67.69% of all human-coded causes of death. No significant differences were observed in these high sensitivity values when comparing periods without excess mortality to periods of excess, severe, and extreme excess mortality. The same holds for specificity, which exceeded 0.96 for all chapters examined, and for the PPV, which surpassed 0.75 in nine chapters, including the more prevalent ones. When considering block classification within the three most common ICD-10 chapters, AUTOCOD maintained high performance, demonstrating high sensitivity (≥0.75) for 13 ICD-10 blocks, high PPV for nine blocks, and specificity above 0.98 in all blocks, with no significant differences between periods without excess mortality and those with excess mortality.

Conclusions:

Our findings indicate that during periods of excess and extreme excess mortality, AUTOCOD's performance remains unaffected by potential text quality degradation due to pressure on health services. Consequently, AUTOCOD can be dependably employed for real-time specific-cause mortality surveillance, even in extreme excess mortality situations.