JMIR Publications

ABSTRACT

Background:

The growing use of artificial intelligence (AI) to support patient portal message management requires rigorous pre-clinical evaluation. Directly testing AI within electronic health record (EHR) systems poses significant safety, workflow, and data-governance risks. We developed and describe a secure user interface (UI) that enables clinical and technical teams to experiment with AI for portal messaging on de-identified data before clinical integration.

Objective:

We developed a secure user interface (UI) that enables clinical and technical teams to experiment with AI for portal messaging on de-identified data before clinical integration, which we describe here for others to adapt.

Methods:

We developed a web UI in Python 3 with a modular backend for data handling, de-identification, and AI task execution. The system runs in a secure research environment equipped with an NVIDIA GRID T4-1Q GPU and institutional access controls. An IRB-approved corpus of patient-portal messages was cleaned to a dementia-relevant subset of 6,941 Medical Advice Request messages. We designed a de-identification pipeline to remove or replace personal health identifiers. The platform supports single-message and batch workflows and exposes exemplar LLM-enabled tasks such as authorship identification, message categorization, criticality flagging, and response drafting using zero-, one-, and few-shot prompting. Only de-identified text is sent to the model endpoint.

Results:

The system successfully executed end-to-end workflows: ingest de-identified messages, run individual or batch AI analyses, and present outputs for review. PHI masking was consistently applied across the corpus. Exemplar runs demonstrated prompting strategies to yield interpretable outputs for authorship identification, categorization, and criticality flagging, while response drafting produced editable clinician starting points. A token-based cost readout provided transparent operating estimates for LLM-backed tasks.

Conclusions:

This framework offers a practical path to study AI behavior on real, de-identified messages without affecting live EHR workflows and thus supports exploratory testing, prompt iteration, and comparative analyses, including LLM prompts versus baseline models, while preserving governance boundaries. We discuss design choices, safety controls, and the limits of a sandbox approach. A secure, UI-based sandbox enables health-system teams to evaluate AI for patient-portal messaging before clinical integration.