JMIR Publications

ABSTRACT

The growing use of artificial intelligence (AI) to support patient-portal message management requires rigorous pre-clinical evaluation. Directly testing AI within electronic health record (EHR) systems poses significant safety, workflow, and data-governance risks. Here, we present a technical feasibility report on a secure user interface (UI) sandbox designed to enable clinical and technical teams to experiment with AI for portal messaging on de-identified data before clinical integration. Here, a sandbox refers to a controlled, non-production environment that allows safe testing, prompt iteration, and evaluation of AI outputs without impacting live EHR systems or patient care. We developed a web UI in Python 3 with a modular backend for data handling, de-identification, and AI task execution. The system runs in a secure research environment equipped with an NVIDIA GRID T4-1Q GPU and institutional access controls. We designed a de-identification pipeline to remove or replace personal health identifiers. The platform supports single-message and batch workflows and exposes exemplar LLM-enabled tasks such as authorship identification, message categorization, criticality flagging, and response drafting using zero-, one-, and few-shot prompting. Only de-identified text is sent to the model endpoint. The system successfully executed end-to-end workflows: ingest de-identified messages, run individual or batch AI analyses, and present outputs for review. PHI masking was consistently applied across the corpus. We ran use cases with an IRB-approved corpus of a dementia-relevant subset of 6,941 patient portal messages categorized as “Medical Advice Requests.” With the support of the user interface, we tested which prompting strategies to yield interpretable outputs for authorship identification, categorization, and criticality flagging, and whether response drafting produced editable clinician starting points. A token-based cost readout provided transparent operating estimates for LLM-backed tasks. This framework offers a practical path to test AI behavior on real, de-identified messages without affecting live EHR workflows and thus supports exploratory testing, prompt iteration, and comparative analyses, including LLM prompts versus baseline models, while preserving governance boundaries. We discuss design choices, safety controls, and the limits of a sandbox approach. A secure, UI-based sandbox enables health-system teams to evaluate AI for patient-portal messaging before clinical integration. The goal is not to assume benefit but to generate evidence about feasibility, risks, and fit to clinical needs in a controlled setting.