Patient portal messaging has expanded rapidly and now encompasses clinical questions, symptom updates, and coordination tasks that can affect care quality and timeliness [1-4]. As message volumes rise, concerns about clinician workload and burnout have grown [2]. In parallel, health systems are exploring AI to help manage message flow through triage support, authorship cues, categorization, and draft responses [5]. Despite optimism that AI may help, its true impact on workload, safety, and equity remains uncertain [6-8].

The increasing use of AI to support patient portal message management requires rigorous testing and evaluation. Patient portal messages convey critical health information, from appointment reminders and laboratory results to educational content, making clarity, accuracy, and accessibility essential. Even minor ambiguities or technical errors in poorly designed AI can lead to misunderstandings and compromise care quality [9-11].

Developing AI tools for portal inbox management is particularly challenging within live electronic health record (EHR) systems. Direct integration at early stages can introduce significant risks, including threats to data integrity and barriers to agile development [12]. The sensitive nature of health information necessitates a secure, isolated environment to test messaging functions without endangering patient safety or system stability. A dedicated testing platform is therefore critical because it must realistically simulate message flows, support the creation and evaluation of dynamic content, and enable real-time interactions with patients and clinicians. Importantly, this environment should remain separate from the EHR during development and testing to allow researchers to gather rich feedback, iterate on message design, and validate functionality in a controlled setting to ensure that only vetted, effective approaches reach the live clinical environment.

A sandbox is a controlled, nonproduction environment that allows safe testing, prompt iteration, and evaluation of AI outputs without impacting live EHR systems or patient care. A preclinical sandbox can enable secure experiments with real messages, which is particularly valuable given the risks and difficulties of direct experimentation inside live EHRs [12]. A well-designed sandbox could separate front-end interactions from back-end processing, provide robust deidentification, allow plug-and-play AI components, and offer transparent telemetry estimates, including latency, token counts, and error traces. Above all, it should keep experimentation out of production until teams understand behavior, constraints, and risks [13]. Building on this need, we present a technical feasibility report describing a user interface (UI)–based sandbox developed within a secure, governed research environment.

The system executed the complete pipeline, from message ingestion to output presentation, without accessing live EHR systems. Across the 6941-message feasibility corpus, the interface supported single-message testing, batch processing, prompt selection, task execution, cost telemetry, and export of results. Because this manuscript reports technical feasibility rather than formal model evaluation, we do not present task-level output distributions or performance metrics.

AI-supported task execution was averaged approximately 5 seconds end-to-end (deidentification plus all 4 AI tasks) per message. The application included retry handling for API rate-limit errors, with up to 3 retry attempts for HTTP 429 responses. Average message length was approximately 150 to 200 words, with prompt-based processing estimated at 300 to 400 input tokens and 100 to 150 output tokens per message. Using conservative pricing assumptions, this corresponded to an estimated upper-end cost of approximately US $0.044 per message. These values are presented as operational feasibility estimates rather than production benchmarks.

We ran deidentification by replacing personal names, dates, and contact details with deterministic placeholders while preserving contextual meaning. Biomedical entity coverage for medication mentions was achieved by combining scispaCy models with a small, curated lexicon. Inspection of deidentified text confirmed that messages remained coherent and suitable for downstream analysis, although occasional overmasking of medication names was observed. To evaluate the NER pipeline used for deidentification, we tested performance on a message set. Manual review of pipeline output was conducted to identify false positives and false negatives, yielding 110 adjudicated entities. The pipeline achieved 82.1% precision, 80.9% accuracy, 95.1% sensitivity, and 39.3% specificity, with 4 false negatives identified. Because the deidentification did not achieve 100% sensitivity, all nonsynthetic messages were treated as containing PHI, including after they underwent deidentification.

In the use cases, we found that authorship prompts produced rational, auditable rationales (eg, references to third-person phrasing or caregiver self-identification) useful for reviewer feedback. We observed failure modes, such as the AI wrongly detecting care partner message authorship when in fact a patient authored the message and mentioned a spouse or family member. We also tested whether categorization and criticality prompts yielded stable labels on representative samples, and whether disagreements across prompting strategies were visible in the UI. Showing the output from the categorization and criticality prompts allowed us to collect feedback from key partners (informatics leads and physicians) that may have been otherwise hard to collect. Similarly, we were able to explore whether response drafting generated concise, editable text. We observed that the model occasionally produced overly generic and lengthy replies that did not directly address the specific clinical concern raised in the message. Batch runs were completed on the 6941-message subset and exported to a results table that combined input metadata, task outputs, and prompt provenance.

The token panel surfaced per-message and aggregate token counts, allowing users to see how different prompt templates affect inference cost. This made trade-offs between few-shot accuracy and budget more tangible during experimentation.

A core challenge for AI in patient portal messaging is that most tools reach clinicians long before they have been studied with realistic data and users [5,19]. Our approach prioritizes a preclinical testbed that allows health system teams to probe behavior, iterate prompts, and examine costs in a controlled setting. Several design choices were important. First, we enforced a privacy boundary: all messages were partially masked before any model access and treated as containing PHI throughout the entire process. Second, we made prompts first-class, versioned objects so that results are reproducible and modifiable. Third, we exposed batch processing alongside single-message testing to support both qualitative review and early quantitative exploration without implying clinical readiness. Our initial corpus was intentionally narrow (dementia-relevant medical advice requests from older adults), which constrains claims about performance across other message types and populations. The UI architecture, however, was built to be extensible: additional message datasets, specialized lexicons, and prompt templates can be slotted in without changing the UI.

The sandbox helps align stakeholders around evidence rather than assumptions by making experimental outputs visible and auditable. Engineers can use the environment to compare prompt templates, routing logic, and cost trade-offs; clinicians can preview which outputs reduce workload and which create new review burden; and governance teams can assess safety and equity risks before any integration decisions. The current implementation uses the OpenAI Python SDK configured for Azure OpenAI within the institutional environment. The AI layer was designed for extensibility, meaning future versions could evaluate baseline classifiers, alternative model end points, or local open-weight models as institutional policies and infrastructure evolve. However, this feasibility study did not demonstrate model substitution or formal head-to-head model comparisons. This manuscript reports a technical feasibility study rather than a formal evaluation. Future work will use this environment to conduct formal validation, comparisons with baseline classifiers and local open-weight models, turnaround times, clinician workflow impact, and staff satisfaction.

This work has limitations. We did not conduct a formal accuracy study or randomized usability trial; our evaluation focused on feasibility, safety controls, and instrumentation. Deidentification quality, while strong in bench tests, requires formal validation against gold standards. While the sensitivity was strong, we continued to treat the messages as containing PHI after the deidentification. The UI was developed and tested in an environment within the institution’s firewall, and nonsynthetic messages were only accessed by IRB-approved team members.

Deidentification was optimized for sensitivity to minimize PHI disclosure risk, accepting lower specificity as a deliberate trade-off. This level of overmasking, in which non-PHI terms, including some medication names, were redacted, may have reduced the semantic richness available to the LLM during preproduction testing and could have affected the quality of generated responses. The extent of this impact was not formally evaluated and represents a limitation of the current study. Medication and identifier coverage depends on biomedical NER supplemented by regex-based detection, curated medication lists, text normalization, and fuzzy matching. Although these steps improve detection of common variants and misspellings, broader use across clinical settings will require periodic updates for new medications, local abbreviations, and institution-specific terminology. The pipeline was designed with HIPAA (Health Insurance Portability and Accountability Act) Safe Harbor identifier categories in mind, but it should not be interpreted as a certified Safe Harbor or Expert Determination implementation. In settings where data would leave the institutional security perimeter, formal validation of deidentification accuracy should be evaluated before using the workflow outside a similarly governed environment.

The UI also lacked automated safeguards, such as confidence thresholds or contradiction detection. Future work will require automated safeguards and explicit strategies for long-thread management, such as truncation, summarization, or controlled context-window selection, for use cases that require reasoning across full conversation histories. Token-based cost views are approximate and will shift with pricing and prompt designs. Finally, because the system deliberately avoids live EHR integration, we did not assess downstream operational effects such as turnaround times or staff satisfaction.

The UI does not yet allow the import of demographic data; thus, we were unable to assess differential performance across demographics. With the added feature of demographic data imports in future versions, this environment could enable assessment of differential performance across demographic characteristics, such as age, race, and ethnicity, to evaluate the presence of bias and equity considerations before deployment.

Despite these limitations, a secure UI-based sandbox is a pragmatic step toward responsible AI for patient messaging. It allows organizations to study real behaviors on real messages securely, identify failure modes early, and decide which use cases, if any, merit the cost and complexity of a full integration.

AI for patient portal messaging is advancing quickly, but its real impact on clinicians, patients, and care partners remains uncertain [20-22]. A secure sandbox with an accessible UI provides a practical way to test and refine AI use cases before they reach clinical workflows. Our implementation demonstrates end-to-end feasibility across the full pipeline: data ingestion, privacy preservation, configurable prompting, batch analysis, and transparent cost telemetry, all within a governed environment. The goal is not to assume benefit but to create the conditions under which benefit, risk, and fit can be studied with rigor.