Data from 350 patients with AAP were retrospectively collected at Jeroen Bosch Hospital, a Dutch teaching hospital, between July 2016 and January 2023. Patient inclusion criteria and cohort construction are detailed in our previous study [18]. This dataset included 167 appendicitis cases with 169 other AAP presentations. Among other AAP presentations, those suspected of appendicitis were balanced with those having nonspecific or other AAP causes based on initial ED assessments by triage nurses or referring physicians. Cases lacking sufficient medical history or physical examination data or missing more than 70% (n=14) of vital signs were excluded, resulting in 336 cases. Each case contained triage and intake data, vital signs, and the medical history and physical examination sections. Further cohort details are in Supplemental Tables 1A-E of our previous publication [18]. No exclusions were made based on age, comorbidities, medication use, or symptom presentation. Data were extracted and pseudonymized using CTcue (IQVIA Nederland B.V.).

All 336 ED reports were independently annotated by 2 lead researchers to establish the reference standard for 16 clinical signs and symptoms, as previously described [18] (Multimedia Appendix 1). Interrater agreement, assessed on a random sample of 80 reports, demonstrated high reliability, with average Krippendorff α values of 0.93 for binary features and 0.95 for the multiclass feature, and an average Jaccard similarity of 0.76 for multilabel features. The 16 features were derived from our previous HIVE model, in which XGBoost (Extreme Gradient Boosting) and SHAP (Shapley Additive Explanations) analyses were used to identify features contributing more than 95% of the model’s predictive output (Multimedia Appendix 2). Their discriminative performance was previously assessed using the area under the receiver operating characteristic curve (AUROC) to determine the optimal feature set. The final selection comprised 11 medical history and 5 physical examination features, covering 8 binary, 1 multiclass, and 7 multilabel outputs.

LLM-based feature extraction using a 0-shot prompting strategy was chosen over conventional tokenization-based text classification because the goal was to extract clinically meaningful variables and integrate them with structured vital signs and intake information directly available from the EHR. Traditional token-based NLP methods focus on identifying or classifying text spans and cannot reliably capture negation, multilabel outputs, or nuanced clinical context, and typically provide token-level outputs that obscure the contribution of specific clinical findings. This reduces interpretability and alters feature-importance patterns (eg, via SHAP), limiting transparency and alignment with the structured design of the HIVE model.

For the present analysis, 100 reports were used to optimize two 0-shot prompting strategies for extracting these features using an LLM. Zero-shot prompting directly instructs the LLM without examples [19]. The remaining 236 reports were reserved to evaluate manual labeling by 2 ED physicians and automated extraction by the LLM (Figure 1A).

These reports were also used for the HIVE appendicitis prediction model. Of the 336 reports, 268 had previously been labeled by the lead researchers and were used for model training and tuning. The remaining 68 reports, drawn from the 236 physician-labeled and LLM-processed reports for evaluation, served as 4 identical validation sets (Figure 1B). This validation set was selected before model development, contained no cases used in LLM prompt construction or hyperparameter tuning, and matched the validation size used in our previous study [18]. This ensured a fully independent and consistent benchmark for assessing the effect of LLM-based feature extraction on downstream model performance. Given the modest dataset size (n=336) and single-center design, this study aimed to evaluate feasibility rather than generalizable performance. The results should therefore be interpreted as proof-of-concept.

The primary objective of this study was to evaluate the LLM’s automated extraction of 16 clinical signs and symptoms from ED reports, all of which contributed to our ML (HIVE) model (Multimedia Appendix 2). To that end, a 4-step workflow was implemented (Figure 2). First, ED reports were compiled into a JSON file. Next, a base prompt was designed, consisting of an instruction specifying which symptom or sign to extract and an output with a predefined set of answer options. From this base prompt, two 0-shot prompting strategies were derived. The minimal prompt strategy extended the base prompt with a limited set of essential clarifications, identical to the annotation rules provided to ED physicians to ensure equal comparison. The optimized prompt strategy further integrated additional elements aimed at steering performance, including broader context (artificial intelligence [AI] persona and report type), additional instructions (section limitations, and negations or constraints of symptoms), and specific context (domain-specific terminology or abbreviations, and symptom explanations). To quantify the contribution of individual elements, an ablation study was conducted at the start of the study on a development set of 100 ED reports. Each element was sequentially added to the base prompt, and performance was compared against the researcher’s annotations. Only elements that improved performance were included in the optimized prompt. Both prompts used the same annotation rules; any differences between them are due to the prompt design, not to different instructions. All prompts and reports were provided to the LLM in Dutch. The LLM processed each report from the development (n=100) and evaluation sets (n=236), producing structured output in JSON format. The JSON format consisted of a pseudonymized patient identifier, the name of the target symptom, and the predefined answer options of the target symptom (eg, {“uid”: “82547948A6A5E91B93B2BAACAE3F943508FD7EFA,” “abdominal pain location”: [“right lower quadrant,” “diffuse”]}). Third, features extracted from 68 of 236 evaluation reports were combined with structured intake data and vital signs. Finally, these datasets were used for ML model validation. The remaining 268 cases, annotated by the research team, were used for developing the ML model.

In December 2024 and January 2025, four popular LLMs available on the Ollama platform were evaluated using 4 representative features varying in complexity and output type. Selected models, Mistral-Nemo 12B Instruct [20], Qwen2.5:14B [21], DeepSeek R1:14B distilled from Qwen2.5 [22], and Gemma2:9B [23], were chosen for their multilingual capabilities, input length, and compact size (<10 GB), allowing deployment on a 12 GB GPU laptop without requiring connection to a computational cluster, which may be impractical in ED settings.

After evaluation, Alibaba’s Qwen2.5:14B Instruct was selected based on performance on the development set (n=100). All LLMs were assessed on the same data, prompts, and settings (eg, temperature=0.1), supporting deterministic output [24]. Ollama was chosen for its capability to facilitate local LLM deployment. The underlying LLM extraction tool for this study is available in a repository (10.6084/m9.figshare.28931030) and on GitHub [25] with a complete walkthrough to adopt the framework and tooling, and incorporates components from a previously published data extraction repository [26].

LLM outputs were checked against the predefined answer options. To mimic daily practice, where manual postprocessing is infeasible, responses outside these options were removed. Occasionally, the LLM returned descriptive terms accurately copied from the ED report but not listed among the predefined answers. These nonconforming terms were removed, and the feature was labeled as “not reported.” All performance metrics presented reflect this postprocessing step.

A reader study was conducted to compare LLM outputs, under the minimal and optimized prompt strategies, with manual labeling of 2 ED physicians using the same evaluation set (n=236). Each ED report was presented in its original format and independently reviewed by 2 ED physicians, one with 2 years of residency experience and one with 5 years of postqualification experience. Physicians labeled 16 clinical signs and symptoms using an Excel (Microsoft Corp) spreadsheet. Predefined answer options matched those given to the LLM and were provided as drop-down menus: single-choice for binary and multiclass features, and multiple-choice for multilabel features. If a sign or symptom was absent or not reported, the field was left blank.

LLM- and physician-labeled features were evaluated using specificity, sensitivity, and accuracy. The HIVE model’s performance was assessed via AUROCs. For multiclass and multilabel features, LLM performance metrics were calculated for each positive class and averaged over all positive classes. 95% CIs were estimated through bootstrapping with replacement.

An XGBoost (version 2.1.1) algorithm was selected to estimate the probability of appendicitis, following the approach described in our previous study [18,27]. Referred to as the HIVE model, it incorporates ED intake information, vital signs, medical history, and physical examination features. As explained in this study’s design, only routinely measured vital signs and intake information, combined with medical history and physical examination features that contributed most to our previous HIVE model, were included in this model. Following the same set-up, 336 cases were divided into 268 cases for training or tuning (researcher-labeled), and 68 validation cases relabeled by the LLM, under 2 prompting strategies, and by 2 ED physicians. This allowed comparison against the original reference annotations.

For model tuning, repeated stratified 10-fold cross-validation was used to preserve class distributions. Categorical parameters were encoded using CatBoost (version 2.6.3) into numerical representations derived from training data [28,29]. Hyperparameters were optimized to maximize AUROC using Bayesian optimization with Optuna (version 3.6.1) over 100 trials [30]. The optimized hyperparameters were a learning rate (η) of 0.012, a maximum tree depth of 3, a minimum sum of weights in a child of 4, a subsample ratio of features for each tree of 0.64, and 156 boosting rounds. Class weights were not adjusted, given that the dataset was approximately balanced (167 appendicitis vs 169 other AAP causes). Feature contributions to model predictions were quantified using TreeSHAP (version 0.46.0). To examine whether downstream performance depended on model choice, we trained a random forest on the same 236 researcher-labeled cases and evaluated it on the same 68-case validation set using the LLM-extracted features. Preprocessing matched the HIVE pipeline, with numeric missing values imputed using an Iterative Imputer (max_iter=10) for compatibility with the random forest model. A standard configuration was used (n_estimators=500, max_depth=None, bootstrap=True). The TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) checklist was followed to ensure methodological transparency (Checklist 1), and all code including the LLM extraction tool are available via figshare (10.6084/m9.figshare.28931030).

This study was conducted according to the Declaration of Helsinki and Guidelines for Good Clinical Practice. The execution of this retrospective observational study of patient records was approved by the local review board of the Jeroen Bosch Hospital (number 2023.11.22.02), and was judged by the Medical Research Ethics Committee Brabant, which waived this study to be subject to the regulations of the Dutch Medical Research Involving Human Subjects Act, including a waiver of informed written consent and a consent for publication (Medical Research Ethics Committee number NW2024-05) [31].

Data were extracted from the EHR using CTcue (IQVIA) and handled in a pseudonymized manner. No directly identifiable patient information was accessed by the researchers. Data extraction and analysis were performed in a secure environment in accordance with institutional and European and national data protection regulations. Participants received no compensation.

Among 336 patients with AAP (median age 41, IQR 22‐62 years, 205/336, 61% were female), 50% (167/336) had appendicitis (Table 1, Table 2). Reports were authored by various physicians over 7 years. Feature prevalence ranged from 6% (n=20) for pollakiuria to 91% (n=303) for pain location (Table 3).

An ablation study on the development set (n=100) quantified the contribution of individual prompt elements (Tables 4-6). Sequentially adding each element to the base prompt showed the largest performance gains for negation or constraint, domain-specific context, and explanation, while section limitation, AI persona, and report type produced smaller and less consistent improvements across features (Tables 5 and 6). These findings are incorporated into the composition of the optimized prompts (Table 4). Some clarifying elements, particularly negation or constraint elements, were also integrated into the minimal prompt to ensure an equal comparison with manual annotations by ED physicians. Certain elements were not applicable to specific features (eg, when no negation or explanation was possible) and are therefore left blank in Tables 5 and 6. Table 7 lists the minimal prompts used to extract each of the 16 features; these incorporate the same annotation rules that were provided to ED physicians. Table 8 presents the corresponding optimized prompts, which integrate the prompt elements identified in the ablation study as improving extraction performance. Multimedia Appendix 3 presents all prompt elements evaluated for each feature in the ablation study.

Table 9 presents the feature-level performance of the LLM under both prompt strategies, evaluated against a reference standard based on lead researcher annotations, shown alongside manual labeling by ED physicians on the evaluation set (n=236). Using the minimal prompt strategy, the LLM achieved an average specificity of 0.929 (95% CI ±0.020), sensitivity of 0.810 (95% CI ±0.058), and accuracy of 0.910 (95% CI ±0.018). Lower feature prevalence in the ED reports did not consistently result in lower LLM accuracy (Table 3). When applying the optimized prompt strategy, performance improved across all metrics; specificity increased to 0.943 (95% CI ±0.017), sensitivity to 0.859 (95% CI ±0.055), and accuracy to 0.929 (95% CI ±0.016). The 5 largest gains in accuracy were observed for nausea, pain manifestation, onset of pain, pain location, and abdominal pain location. No gains appeared for rebound tenderness, anorexia, pain migration to the right lower quadrant, fever, and palpation tenderness. Optimized prompts were particularly beneficial for features involving domain-specific terminology, abbreviations, or overlapping definitions, such as nausea and vomiting.

Both physicians achieved higher weighted average specificity and accuracy on the evaluation set than the LLM under either prompting strategy. Physician 1 reached a specificity of 0.978 (95% CI ±0.011), sensitivity of 0.859 (95% CI ±0.015), and accuracy of 0.961 (95% CI ±0.012), while physician 2 achieved a specificity of 0.980 (95% CI ±0.008), sensitivity of 0.793 (95% CI ±0.056), and accuracy of 0.951 (95% CI ±0.015; Table 2).

At the feature level, the LLM using the optimized prompt matched or exceeded the accuracy of physician 1 for 8 of 16 features, and for physician 2 for 9 of 16. Notably, both physicians outperformed the LLM on the onset of pain, migration to the right lower quadrant, and pain manifestation. For the remaining features, physician performance was slightly higher, with marginal accuracy differences. All feature-level outputs for this comparison are available in Multimedia Appendix 4.

To compare predictive performance using LLM- vs physician-labeled input features, the HIVE model was evaluated for its ability to distinguish appendicitis from other AAP causes on the validation set (n=68; Figure 3). Using LLM-extracted features, the HIVE model achieved an AUROC of 0.871 (95% CI ±0.019) with the minimal prompt, and 0.911 (95% CI ±0.014) with the optimized prompt. In comparison, AUROCs were 0.917 (95% CI ±0.015) using features labeled by physician 1, and 0.924 (95% CI ±0.018) using those from physician 2.

SHAP analysis demonstrated consistent feature importance across all input types (Figure 4). Features are displayed in descending order of their mean absolute SHAP contributions. The overall SHAP distributions were nearly identical, with the top 10 features cumulatively contributing 85% to 85.9% of the model output (full list in Multimedia Appendix 5). Although the minimal 0-shot prompt achieved a lower AUROC of 0.871 (95% CI ±0.019), its feature importance pattern closely resembled those of the optimized and physician-labeled inputs.

The random forest performed similarly to the XGBoost-based HIVE model, with AUROCs of 0.909 (95% CI ±0.13) using the optimized prompt strategy and 0.883 (95% CI ±0.13) using the minimal prompt strategy (Multimedia Appendix 6).

To examine misclassifications, 8 evaluation cases from the 4 lowest-performing features, pain manifestation, complaint development, pain onset, and abdominal-pain location, were reviewed to illustrate diverse error types.

The first 4 cases in Figure 5 highlight challenges with multilabel outputs, where the LLM was expected to report a combination of applicable labels but instead provided only a single answer. Specifically, in cases 1 and 2, pain manifestation evolved from continuous to attack-wise, yet the LLM recognized only 1 label. Similarly, in cases 3 and 4, the LLM missed fluctuating symptom development over time. In Figure 6, cases 5 and 6 illustrate 2 reports lacking explicit mention of the onset of pain; the LLM inferred labels such as “acute” or “gradual,” whereas reference was “not reported.” In case 7, the ED report described a punctum maximum located 4 cm to the right of the umbilicus. While this was labeled as “periumbilical” in the reference standard, the LLM returned “right lower quadrant.” In case 8, despite instructions to report only the primary pain location, the LLM provided both “entire right abdomen” and “diffuse” for a case describing diffuse abdominal pain with pronounced tenderness in the right hemiabdomen.

Each panel presents a snippet from an ED report alongside the corresponding LLM output. Words highlighted in magenta within the ED report indicate incorrect (false negative or false positive) or missed interpretations by the LLM, while words in green represent correct (true positive) interpretations. On the right-hand side, the LLM’s JSON output displays the unique patient identifier, the target clinical feature (magenta), and the extracted output (green). All prompts and ED reports were provided to the LLM in Dutch and have been translated into English for readability.