JMIR Publications

ABSTRACT

Background:

Causality assessment is central to pharmacovigilance but remains resource-intensive and subjective. The applicability of large language models (LLMs) to formal World Health Organization–Uppsala Monitoring Centre (WHO–UMC) drug–adverse event (AE) causality assessment has not been well established.

Objective:

This study aimed to evaluate the performance of LLMs in WHO–UMC causality assessment.

Methods:

A curated set of 55 cases derived from the U.S. FDA Adverse Event Reporting System (FAERS), comprising 337 drug-level assessments, were constructed. Cases involving 2 to 11 suspected drugs were stratified by drug count, and five cases were sampled per stratum. To ensure representation of rare but clinically important categories, five additional cases containing at least one ‘Certain’ drug–AE pair were included. Case data were reorganized into a standardized semi-structured format preserving key elements required for WHO–UMC causality assessment. Domain experts conducted a pilot evaluation to align interpretation criteria prior to independently assessing the final dataset, yielding an inter-expert agreement (Fleiss’ kappa) of 0.762 across 337 drug-level assessments. Multiple prompting strategies, including standard prompting, chain-of-thought (CoT), CoT with self-consistency (CoT-SC), few-shot, reasoning and acting (ReAct), and tree-of-thought (ToT), were applied across multiple LLMs, including GPT-5.4 (and mini variant) and Gemini 2.5 (Flash and Pro), via their respective application programming interfaces (APIs). Agreement with expert assessments was quantified using Cohen’s kappa, weighted kappa, and accuracy metrics. Internal consistency across repeated inferences was evaluated using Fleiss’ kappa.

Results:

Performance varied across models and prompting strategies. Cohen’s kappa ranged from 0.368 to 0.641, while weighted kappa ranged from 0.641 to 0.821. Accuracy ranged from 0.583 to 0.804, and balanced accuracy from 0.513 to 0.735. Fleiss’ kappa ranged from 0.730 to 0.915, corresponding to substantial to almost perfect agreement. Among the evaluated configurations, Gemini 2.5 Flash with CoT-SC prompting showed the highest observed point estimates across several metrics (Cohen’s kappa: 0.640; weighted kappa: 0.821; accuracy: 0.804; balanced accuracy: 0.723; Fleiss’ kappa: 0.915), although the gains over other prompting strategies were modest. Category-level performance for this model showed higher performance for ‘Certain’ (F1-score: 0.793), ‘Probable/Likely’ (F1-score: 0.794), and ‘Unlikely’ (F1-score: 0.898), whereas performance for ‘Possible’ remained substantially lower (F1-score: 0.293), reflecting the difficulty of intermediate causality assessment.

Conclusions:

LLMs demonstrate moderate to substantial agreement in WHO–UMC causality assessment, indicating meaningful but still limited performance relative to expert judgment. While not suitable for independent decision-making, they may serve as supportive tools in pharmacovigilance workflows, particularly for preliminary case triage. Further studies using larger and more diverse datasets and evaluating performance on raw narrative reports are warranted.