Large Language Model Automated Extraction of Clinical Signs and Symptoms From Emergency Department Reports for Machine Learning Prediction Models: Development and Validation Study

Introduction

In the emergency department (ED), clinical information is primarily documented as free-text. While this documentation style aligns with routine clinical practice, it presents challenges for reusing data in research and decision support applications [1-5]. Structured templates in the electronic health record (EHR) system offer a common workaround, but impose extra workload on already time-pressed ED physicians and are rarely adopted in practice [6,7]. Automated feature extraction using natural language processing (NLP) offers a more sustainable alternative. To date, extracting clinical data from ED reports has primarily relied on long short-term memory and BERT (Bidirectional Encoder Representations From Transformers) models, which perform reasonably well but require extensive preprocessing, large manually annotated corpora, and model retraining [8-10].

Generative pretrained large language models (LLMs) present an alternative, as they can be optimized using only prompting, making them more accessible for downstream tasks. While these LLMs have proved effective in extracting diverse features from radiology and pathology reports [11-13], their use in automated feature extraction from ED reports is limited. Unlike radiology or pathology reports, ED reports contain extensive abbreviations, succinct phrasing, and domain-specific terminology, presenting unique challenges to NLP. Recent work has begun to explore LLM-based information extraction in emergency medicine. McMurry et al [14] demonstrated LLM-based identification of binary respiratory symptoms using 0-shot prompting and multicenter validation, while Bejan et al [15] phenotyped symptomatic kidney-stone presentations using a range of prompting and fine-tuning strategies. Gao et al [16] used LLM-generated severity scores, supported by in-context learning and retrieval-augmented generation, to improve early triage predictions. Collectively, recent studies demonstrate the potential of LLM-based feature extraction from ED reports. However, these efforts primarily addressed binary outcomes or limited symptom sets, or did not investigate the downstream impact on diagnostic model performance. Prior work also relied mainly on large LLMs or Chinese bilingual LLMs, leaving open whether smaller multilingual LLMs can perform reliably in underrepresented languages [17]. This is particularly relevant for the Dutch, where the inherent complexity of ED reports is compounded by limited representation in LLM training corpora.

This study addresses these gaps by evaluating whether a small multilingual LLM (Qwen 2.5:14B) can automatically extract binary, multiclass, and multilabel clinical features from Dutch ED reports and provide reliable inputs for a downstream prediction model. Our focus is a clinically relevant use case: acute abdominal pain (AAP). Extracted features are used as input for the HIVE (History, Intake, Vitals, Examination) model, a previously established appendicitis prediction model based on 16 contributing clinical signs and symptoms [18]. These features were originally annotated manually, a process that is labor-intensive and difficult to scale. This study evaluates whether an LLM can automate this extraction and produce comparable inputs by developing and validating two 0-shot prompting strategies. Secondary outcomes included comparison to physician annotations and assessing the appendicitis prediction model using LLM-extracted vs physician-annotated features. We hypothesize that, with appropriate prompting, a small locally run multilingual LLM could achieve near-expert accuracy in feature extraction and maintain comparable predictive models’ performance, supporting the feasibility of a scalable, privacy-preserving workflow for decision support systems at the ED.

Methods

Data Collection

Data from 350 patients with AAP were retrospectively collected at Jeroen Bosch Hospital, a Dutch teaching hospital, between July 2016 and January 2023. Patient inclusion criteria and cohort construction are detailed in our previous study [18]. This dataset included 167 appendicitis cases with 169 other AAP presentations. Among other AAP presentations, those suspected of appendicitis were balanced with those having nonspecific or other AAP causes based on initial ED assessments by triage nurses or referring physicians. Cases lacking sufficient medical history or physical examination data or missing more than 70% (n=14) of vital signs were excluded, resulting in 336 cases. Each case contained triage and intake data, vital signs, and the medical history and physical examination sections. Further cohort details are in Supplemental Tables 1A-E of our previous publication [18]. No exclusions were made based on age, comorbidities, medication use, or symptom presentation. Data were extracted and pseudonymized using CTcue (IQVIA Nederland B.V.).

Study Design

All 336 ED reports were independently annotated by 2 lead researchers to establish the reference standard for 16 clinical signs and symptoms, as previously described [18] (Multimedia Appendix 1). Interrater agreement, assessed on a random sample of 80 reports, demonstrated high reliability, with average Krippendorff α values of 0.93 for binary features and 0.95 for the multiclass feature, and an average Jaccard similarity of 0.76 for multilabel features. The 16 features were derived from our previous HIVE model, in which XGBoost (Extreme Gradient Boosting) and SHAP (Shapley Additive Explanations) analyses were used to identify features contributing more than 95% of the model’s predictive output (Multimedia Appendix 2). Their discriminative performance was previously assessed using the area under the receiver operating characteristic curve (AUROC) to determine the optimal feature set. The final selection comprised 11 medical history and 5 physical examination features, covering 8 binary, 1 multiclass, and 7 multilabel outputs.

LLM-based feature extraction using a 0-shot prompting strategy was chosen over conventional tokenization-based text classification because the goal was to extract clinically meaningful variables and integrate them with structured vital signs and intake information directly available from the EHR. Traditional token-based NLP methods focus on identifying or classifying text spans and cannot reliably capture negation, multilabel outputs, or nuanced clinical context, and typically provide token-level outputs that obscure the contribution of specific clinical findings. This reduces interpretability and alters feature-importance patterns (eg, via SHAP), limiting transparency and alignment with the structured design of the HIVE model.

For the present analysis, 100 reports were used to optimize two 0-shot prompting strategies for extracting these features using an LLM. Zero-shot prompting directly instructs the LLM without examples [19]. The remaining 236 reports were reserved to evaluate manual labeling by 2 ED physicians and automated extraction by the LLM (Figure 1A).

These reports were also used for the HIVE appendicitis prediction model. Of the 336 reports, 268 had previously been labeled by the lead researchers and were used for model training and tuning. The remaining 68 reports, drawn from the 236 physician-labeled and LLM-processed reports for evaluation, served as 4 identical validation sets (Figure 1B). This validation set was selected before model development, contained no cases used in LLM prompt construction or hyperparameter tuning, and matched the validation size used in our previous study [18]. This ensured a fully independent and consistent benchmark for assessing the effect of LLM-based feature extraction on downstream model performance. Given the modest dataset size (n=336) and single-center design, this study aimed to evaluate feasibility rather than generalizable performance. The results should therefore be interpreted as proof-of-concept.

‎

Workflow and Prompting Strategies

The primary objective of this study was to evaluate the LLM’s automated extraction of 16 clinical signs and symptoms from ED reports, all of which contributed to our ML (HIVE) model (Multimedia Appendix 2). To that end, a 4-step workflow was implemented (Figure 2). First, ED reports were compiled into a JSON file. Next, a base prompt was designed, consisting of an instruction specifying which symptom or sign to extract and an output with a predefined set of answer options. From this base prompt, two 0-shot prompting strategies were derived. The minimal prompt strategy extended the base prompt with a limited set of essential clarifications, identical to the annotation rules provided to ED physicians to ensure equal comparison. The optimized prompt strategy further integrated additional elements aimed at steering performance, including broader context (artificial intelligence [AI] persona and report type), additional instructions (section limitations, and negations or constraints of symptoms), and specific context (domain-specific terminology or abbreviations, and symptom explanations). To quantify the contribution of individual elements, an ablation study was conducted at the start of the study on a development set of 100 ED reports. Each element was sequentially added to the base prompt, and performance was compared against the researcher’s annotations. Only elements that improved performance were included in the optimized prompt. Both prompts used the same annotation rules; any differences between them are due to the prompt design, not to different instructions. All prompts and reports were provided to the LLM in Dutch. The LLM processed each report from the development (n=100) and evaluation sets (n=236), producing structured output in JSON format. The JSON format consisted of a pseudonymized patient identifier, the name of the target symptom, and the predefined answer options of the target symptom (eg, {“uid”: “82547948A6A5E91B93B2BAACAE3F943508FD7EFA,” “abdominal pain location”: [“right lower quadrant,” “diffuse”]}). Third, features extracted from 68 of 236 evaluation reports were combined with structured intake data and vital signs. Finally, these datasets were used for ML model validation. The remaining 268 cases, annotated by the research team, were used for developing the ML model.

‎

LLM Selection

In December 2024 and January 2025, four popular LLMs available on the Ollama platform were evaluated using 4 representative features varying in complexity and output type. Selected models, Mistral-Nemo 12B Instruct [20], Qwen2.5:14B [21], DeepSeek R1:14B distilled from Qwen2.5 [22], and Gemma2:9B [23], were chosen for their multilingual capabilities, input length, and compact size (<10 GB), allowing deployment on a 12 GB GPU laptop without requiring connection to a computational cluster, which may be impractical in ED settings.

After evaluation, Alibaba’s Qwen2.5:14B Instruct was selected based on performance on the development set (n=100). All LLMs were assessed on the same data, prompts, and settings (eg, temperature=0.1), supporting deterministic output [24]. Ollama was chosen for its capability to facilitate local LLM deployment. The underlying LLM extraction tool for this study is available in a repository (10.6084/m9.figshare.28931030) and on GitHub [25] with a complete walkthrough to adopt the framework and tooling, and incorporates components from a previously published data extraction repository [26].

Postprocessing of LLM Outputs

LLM outputs were checked against the predefined answer options. To mimic daily practice, where manual postprocessing is infeasible, responses outside these options were removed. Occasionally, the LLM returned descriptive terms accurately copied from the ED report but not listed among the predefined answers. These nonconforming terms were removed, and the feature was labeled as “not reported.” All performance metrics presented reflect this postprocessing step.

Reader Study–Manual Labeling

A reader study was conducted to compare LLM outputs, under the minimal and optimized prompt strategies, with manual labeling of 2 ED physicians using the same evaluation set (n=236). Each ED report was presented in its original format and independently reviewed by 2 ED physicians, one with 2 years of residency experience and one with 5 years of postqualification experience. Physicians labeled 16 clinical signs and symptoms using an Excel (Microsoft Corp) spreadsheet. Predefined answer options matched those given to the LLM and were provided as drop-down menus: single-choice for binary and multiclass features, and multiple-choice for multilabel features. If a sign or symptom was absent or not reported, the field was left blank.

Statistical Analysis

LLM- and physician-labeled features were evaluated using specificity, sensitivity, and accuracy. The HIVE model’s performance was assessed via AUROCs. For multiclass and multilabel features, LLM performance metrics were calculated for each positive class and averaged over all positive classes. 95% CIs were estimated through bootstrapping with replacement.

ML Model Development and Validation

An XGBoost (version 2.1.1) algorithm was selected to estimate the probability of appendicitis, following the approach described in our previous study [18,27]. Referred to as the HIVE model, it incorporates ED intake information, vital signs, medical history, and physical examination features. As explained in this study’s design, only routinely measured vital signs and intake information, combined with medical history and physical examination features that contributed most to our previous HIVE model, were included in this model. Following the same set-up, 336 cases were divided into 268 cases for training or tuning (researcher-labeled), and 68 validation cases relabeled by the LLM, under 2 prompting strategies, and by 2 ED physicians. This allowed comparison against the original reference annotations.

For model tuning, repeated stratified 10-fold cross-validation was used to preserve class distributions. Categorical parameters were encoded using CatBoost (version 2.6.3) into numerical representations derived from training data [28,29]. Hyperparameters were optimized to maximize AUROC using Bayesian optimization with Optuna (version 3.6.1) over 100 trials [30]. The optimized hyperparameters were a learning rate (η) of 0.012, a maximum tree depth of 3, a minimum sum of weights in a child of 4, a subsample ratio of features for each tree of 0.64, and 156 boosting rounds. Class weights were not adjusted, given that the dataset was approximately balanced (167 appendicitis vs 169 other AAP causes). Feature contributions to model predictions were quantified using TreeSHAP (version 0.46.0). To examine whether downstream performance depended on model choice, we trained a random forest on the same 236 researcher-labeled cases and evaluated it on the same 68-case validation set using the LLM-extracted features. Preprocessing matched the HIVE pipeline, with numeric missing values imputed using an Iterative Imputer (max_iter=10) for compatibility with the random forest model. A standard configuration was used (n_estimators=500, max_depth=None, bootstrap=True). The TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) checklist was followed to ensure methodological transparency (Checklist 1), and all code including the LLM extraction tool are available via figshare (10.6084/m9.figshare.28931030).

Ethical Considerations

This study was conducted according to the Declaration of Helsinki and Guidelines for Good Clinical Practice. The execution of this retrospective observational study of patient records was approved by the local review board of the Jeroen Bosch Hospital (number 2023.11.22.02), and was judged by the Medical Research Ethics Committee Brabant, which waived this study to be subject to the regulations of the Dutch Medical Research Involving Human Subjects Act, including a waiver of informed written consent and a consent for publication (Medical Research Ethics Committee number NW2024-05) [31].

Data were extracted from the EHR using CTcue (IQVIA) and handled in a pseudonymized manner. No directly identifiable patient information was accessed by the researchers. Data extraction and analysis were performed in a secure environment in accordance with institutional and European and national data protection regulations. Participants received no compensation.

Results

Overview

Among 336 patients with AAP (median age 41, IQR 22‐62 years, 205/336, 61% were female), 50% (167/336) had appendicitis (Table 1, Table 2). Reports were authored by various physicians over 7 years. Feature prevalence ranged from 6% (n=20) for pollakiuria to 91% (n=303) for pain location (Table 3).

Ablation Study

An ablation study on the development set (n=100) quantified the contribution of individual prompt elements (Tables 4-6). Sequentially adding each element to the base prompt showed the largest performance gains for negation or constraint, domain-specific context, and explanation, while section limitation, AI persona, and report type produced smaller and less consistent improvements across features (Tables 5 and 6). These findings are incorporated into the composition of the optimized prompts (Table 4). Some clarifying elements, particularly negation or constraint elements, were also integrated into the minimal prompt to ensure an equal comparison with manual annotations by ED physicians. Certain elements were not applicable to specific features (eg, when no negation or explanation was possible) and are therefore left blank in Tables 5 and 6. Table 7 lists the minimal prompts used to extract each of the 16 features; these incorporate the same annotation rules that were provided to ED physicians. Table 8 presents the corresponding optimized prompts, which integrate the prompt elements identified in the ablation study as improving extraction performance. Multimedia Appendix 3 presents all prompt elements evaluated for each feature in the ablation study.

Minimal vs Optimized Prompt Strategies

Table 9 presents the feature-level performance of the LLM under both prompt strategies, evaluated against a reference standard based on lead researcher annotations, shown alongside manual labeling by ED physicians on the evaluation set (n=236). Using the minimal prompt strategy, the LLM achieved an average specificity of 0.929 (95% CI ±0.020), sensitivity of 0.810 (95% CI ±0.058), and accuracy of 0.910 (95% CI ±0.018). Lower feature prevalence in the ED reports did not consistently result in lower LLM accuracy (Table 3). When applying the optimized prompt strategy, performance improved across all metrics; specificity increased to 0.943 (95% CI ±0.017), sensitivity to 0.859 (95% CI ±0.055), and accuracy to 0.929 (95% CI ±0.016). The 5 largest gains in accuracy were observed for nausea, pain manifestation, onset of pain, pain location, and abdominal pain location. No gains appeared for rebound tenderness, anorexia, pain migration to the right lower quadrant, fever, and palpation tenderness. Optimized prompts were particularly beneficial for features involving domain-specific terminology, abbreviations, or overlapping definitions, such as nausea and vomiting.

LLM vs Physician Labeling

Both physicians achieved higher weighted average specificity and accuracy on the evaluation set than the LLM under either prompting strategy. Physician 1 reached a specificity of 0.978 (95% CI ±0.011), sensitivity of 0.859 (95% CI ±0.015), and accuracy of 0.961 (95% CI ±0.012), while physician 2 achieved a specificity of 0.980 (95% CI ±0.008), sensitivity of 0.793 (95% CI ±0.056), and accuracy of 0.951 (95% CI ±0.015; Table 2).

At the feature level, the LLM using the optimized prompt matched or exceeded the accuracy of physician 1 for 8 of 16 features, and for physician 2 for 9 of 16. Notably, both physicians outperformed the LLM on the onset of pain, migration to the right lower quadrant, and pain manifestation. For the remaining features, physician performance was slightly higher, with marginal accuracy differences. All feature-level outputs for this comparison are available in Multimedia Appendix 4.

HIVE Model Performance Using LLM vs Physicians’ Inputs

To compare predictive performance using LLM- vs physician-labeled input features, the HIVE model was evaluated for its ability to distinguish appendicitis from other AAP causes on the validation set (n=68; Figure 3). Using LLM-extracted features, the HIVE model achieved an AUROC of 0.871 (95% CI ±0.019) with the minimal prompt, and 0.911 (95% CI ±0.014) with the optimized prompt. In comparison, AUROCs were 0.917 (95% CI ±0.015) using features labeled by physician 1, and 0.924 (95% CI ±0.018) using those from physician 2.

‎

SHAP analysis demonstrated consistent feature importance across all input types (Figure 4). Features are displayed in descending order of their mean absolute SHAP contributions. The overall SHAP distributions were nearly identical, with the top 10 features cumulatively contributing 85% to 85.9% of the model output (full list in Multimedia Appendix 5). Although the minimal 0-shot prompt achieved a lower AUROC of 0.871 (95% CI ±0.019), its feature importance pattern closely resembled those of the optimized and physician-labeled inputs.

‎

The random forest performed similarly to the XGBoost-based HIVE model, with AUROCs of 0.909 (95% CI ±0.13) using the optimized prompt strategy and 0.883 (95% CI ±0.13) using the minimal prompt strategy (Multimedia Appendix 6).

Error Analysis LLM

To examine misclassifications, 8 evaluation cases from the 4 lowest-performing features, pain manifestation, complaint development, pain onset, and abdominal-pain location, were reviewed to illustrate diverse error types.

The first 4 cases in Figure 5 highlight challenges with multilabel outputs, where the LLM was expected to report a combination of applicable labels but instead provided only a single answer. Specifically, in cases 1 and 2, pain manifestation evolved from continuous to attack-wise, yet the LLM recognized only 1 label. Similarly, in cases 3 and 4, the LLM missed fluctuating symptom development over time. In Figure 6, cases 5 and 6 illustrate 2 reports lacking explicit mention of the onset of pain; the LLM inferred labels such as “acute” or “gradual,” whereas reference was “not reported.” In case 7, the ED report described a punctum maximum located 4 cm to the right of the umbilicus. While this was labeled as “periumbilical” in the reference standard, the LLM returned “right lower quadrant.” In case 8, despite instructions to report only the primary pain location, the LLM provided both “entire right abdomen” and “diffuse” for a case describing diffuse abdominal pain with pronounced tenderness in the right hemiabdomen.

Each panel presents a snippet from an ED report alongside the corresponding LLM output. Words highlighted in magenta within the ED report indicate incorrect (false negative or false positive) or missed interpretations by the LLM, while words in green represent correct (true positive) interpretations. On the right-hand side, the LLM’s JSON output displays the unique patient identifier, the target clinical feature (magenta), and the extracted output (green). All prompts and ED reports were provided to the LLM in Dutch and have been translated into English for readability.

‎

‎

Discussion

Conventional ML prediction models typically require structured input, posing a significant barrier to clinical integration, especially in EDs where physicians predominantly use free-text reports. This study demonstrates that a locally deployable LLM can automatically extract clinically relevant signs and symptoms from unstructured text, including binary, multiclass, and multilabel features. Using an optimized prompt strategy, the LLM reliably extracted features from free-text ED reports, enabling the appendicitis prediction model to achieve diagnostic performance comparable to that based on manually labeled features by ED physicians. This performance was consistent with that achieved in our previous study [18], underscoring the reliability of the LLM in this context.

Numerous ML prediction models have relied on clinical signs and symptoms to support early disease detection and guide ED workflows. While NLP techniques are commonly used to structure this data, each comes with limitations. Conventional techniques, such as bag-of-words, term frequency-inverse document frequency, and phrase skip-gram, rely on word frequency or co-occurrence, with limited ability to capture contextual meaning [32-36]. As a result, they often miss subtle language cues, such as symptom affirmation, negation, or uncertainty, which are essential in clinical interpretation. Topic modeling can uncover broad textual themes but may not align well with specific clinical concepts [33]. Moreover, these techniques are unsupervised, meaning they do not rely on labeled data, which makes their output more difficult to interpret and control. Deep learning models used in this context (eg, long short-term memory or BERT) understand the meaning of words in a broader context [37-40] but require fine-tuning to handle domain-specific language, large annotated datasets, and are challenged by imbalanced data distributions, limiting their performance and generalizability in real-world settings [41,42].

A small, locally hosted LLM for feature extraction can be deployed with modest requirements, offering advantages in data security and independence from external infrastructure. In comparison with conventional NLP or deep learning models, the LLM used in this study required neither extensive preprocessing nor fine-tuning nor large annotated datasets. Instead, it successfully extracted relevant features directly from raw text using relatively simple 0-shot prompting strategies. Importantly, the LLM was used solely for feature extraction; diagnostic predictions were generated by a separate ML prediction model. This modular approach preserves interpretability and statistical transparency, including calibrated probabilities and CIs. SHAP analysis confirmed that the model’s internal reasoning remained stable across input types: AUROC differences primarily reflected feature extraction accuracy rather than alterations in model reasoning. A random forest benchmark produced similar AUROCs to the XGBoost algorithm, indicating that the predictive signal in the LLM-extracted features is consistent across ML algorithms and not dependent on a specific model choice. Although the approach is lightweight and easy to deploy, real-world use still requires seamless EHR integration and fast inference. Modern EHR systems increasingly provide vendor APIs and Python-based toolkits for AI integration, but embedding such workflows into routine EHR practice remains a significant challenge [43]. Given the diagnostic complexity of appendicitis [44-46], this workflow could function as a safety net by flagging high-risk presentations and supporting diagnostic decisions. Larger multicenter evaluations, including physician performance with and without AI assistance, will be needed to determine real-world value.

The ablation study showed that adding specific, symptom-related context, including explanatory cues, domain-specific terminology, and negation or constraint handling, most consistently improved extraction performance. These elements enhanced the LLM’s ability to interpret synonyms and abbreviations commonly used in Dutch ED reports (eg, “misselijkheid,” “nausea,” or “N+” for nausea present) and to distinguish related but distinct concepts (eg, ignore vomiting when assessing nausea). In contrast, broader context elements such as report type, AI persona, or section limitation contributed little to performance. The LLM generalized well across features, classes, and prevalence levels, without requiring balancing strategies, and the close agreement with physician annotations suggests that more complex prompting strategies (eg, few-shot or chain-of-thought) add limited value for this task.

Despite its strengths, error analysis revealed that the LLM struggled with multilabel features, particularly when symptoms evolve over time. Clearly and consistently described features such as nausea or stool consistency were extracted more reliably than the onset of pain, which the LLM may have inferred from the surrounding context. In some cases, the LLM failed to prioritize the most prominent symptom, possibly due to reliance on textual similarity over clinical meaning, as observed in determining abdominal pain location. The LLM occasionally returned descriptive terms that matched the ED report but not the prescribed answer options. Some discrepancies also reflected inherent ambiguities between LLM output and manual annotations. Importantly, the LLM did not hallucinate nonexistent features outside the instructed features and answer options. These findings highlight the potential of LLM-based feature extraction, but also have limitations in handling time discrepancies or ambiguity, which are common challenges in clinical narratives.

This study has several limitations. Only 1 LLM (Qwen 2.5:14B) was evaluated, and performance may differ across models with different architectures or training corpora. As LLMs evolve, future LLMs with improved language understanding or instruction-tuned capabilities may outperform current results and reduce the need for prompt optimization. Nonetheless, given the complexity of ED reports, including abbreviations, medical jargon, and ambiguous phrasing, a structured prompting framework, as presented in this study, will likely remain necessary to ensure consistency and clinically accurate outputs. This study focused on Dutch-language ED reports, reflecting the clinical and linguistic context of our setting. While the prompting framework may be transferable, prompt content must be tailored to language-specific clinical conventions [47]. Translating Dutch reports would not serve as a valid proxy for cross-lingual evaluation because it measures translation quality rather than the LLM’s ability to process native clinical text. Although most open-source multilingual LLMs perform best in English, Chinese, and Spanish [17,48,49], the LLM in this study achieved strong results in Dutch, suggesting that carefully designed prompts can support effective use in underrepresented languages. Despite the single-center scope, the dataset spans 7 years and includes reports from many ED physicians with varying documentation styles, supporting robustness, but external validation across linguistic and clinical contexts is needed [50]. Finally, this study focused on extracting signs and symptoms, not numeric values as laboratory results or vital signs, which are typically stored in structured EHR fields.

This study demonstrates that a locally deployable LLM can automatically extract clinical signs and symptoms from free-text Dutch ED reports with performance comparable to manual labeling by two ED physicians using a structured prompt framework, even in a less widely represented language. This approach has the potential to support scalable implementation of our ML prediction models across clinical contexts, without extensive human annotation or computational resources. Ultimately, it may inform future development of real-time integration of prediction models into clinical workflows, facilitating more scalable, transparent, and privacy-preserving decision-support systems.