JMIR Publications

ABSTRACT

Background:

Postoperative nausea and vomiting (PONV) is a common complication after anesthesia. However, vomiting represents a clinically distinct and objectively measurable endpoint.

Objective:

This study aimed to develop and internally validate predictive models for postoperative vomiting within 24 hours (POV 24h) using structured perioperative data and unstructured clinical text, while introducing a structured framework that separates feature construction from interpretability using large language models (LLMs).

Methods:

We analyzed 33,460 anesthesia records from a single center (2019–2022). Two temporally defined prediction tasks were constructed to reflect real-world clinical decision-making and prevent information leakage: a preoperative model using variables available before anesthesia induction, and a perioperative model using variables available up to the end of surgery. Structured data were modeled using machine learning algorithms (Logistic Regression, Random Forest, XGBoost, LightGBM). Unstructured clinical text was incorporated through a deterministic, concept-driven preprocessing pipeline, where LLMs were used solely for normalization (temperature = 0) without feature generation, followed by rule-based concept mapping and feature encoding. Post-hoc interpretability was further supported using an LLM-based QAChain module. Model performance was evaluated using ROC-AUC, PR-AUC, calibration metrics (Brier score), and threshold-based measures.

Results:

A total of 33,460 surgeries were included, with 3,607 POV events (10.8%). The preoperative model achieved ROC-AUC of 0.7173 and PR-AUC of 0.2049, while the perioperative model achieved ROC-AUC of 0.7200 and PR-AUC of 0.2095. Calibration analysis showed Brier scores of 0.0897 and 0.0833, respectively. Incorporating text-derived features provided modest improvements, while LLM-based explanation modules enhanced interpretability without substantially improving predictive performance.

Conclusions:

Machine learning models can effectively predict postoperative vomiting within 24 hours using perioperative data. The proposed framework demonstrates that LLMs can be integrated in a controlled and reproducible manner—restricted to deterministic normalization and post-hoc reasoning—thereby enhancing interpretability without introducing information leakage or altering predictive modeling. This design supports clinically grounded, transparent decision-making in perioperative risk assessment.