We conducted a multicenter retrospective study using electronic health record narratives, antibiotic orders (including prescription elements such as dose and route), and biochemical laboratory reports from hospitalized patients with pneumonia at 2 hospitals in China. Beijing Tsinghua Changgung Hospital served as the development cohort (n=233; January 2020-January 2024) and was split into training and internal test sets using a 7:3 ratio. Beijing Friendship Hospital served as the external validation cohort (n=98; April 2019-September 2019).

Inclusion criteria were as follows: (1) an admission diagnosis of community-acquired or hospital-acquired pneumonia, (2) complete clinical records, and (3) at least one antibiotic order during hospitalization. The exclusion criterion was confirmed COVID-19 pneumonia.

We constructed the dataset for model input and evaluation using three data domains: (1) patient context, including admission narratives such as the chief concern, history of present illness, physical examination, and initial diagnosis; (2) medication orders, including antibiotic prescriptions and related elements such as drug name, dose, route, and dosing frequency where available; and (3) laboratory indicators, including 7 routinely used hepatic and renal function indicators closely related to antimicrobial safety (Table S1 in Multimedia Appendix 1), which were used to support dose recommendation and safety assessment.

For evaluation, raw antibiotic orders were not directly used as the gold standard but served as source information for reference label construction. Two experienced respiratory physicians reviewed the admission records and laboratory results to determine the final reference regimens. Therefore, model performance was evaluated against expert-adjudicated reference regimens rather than unreviewed raw electronic health record antibiotic orders.

To improve the safety and clinical compliance of the recommendations, we incorporated clinician-defined rule constraints curated by physicians. These rules covered care setting and severity stratification, pathogen risk and coverage strategies, restrictions and contraindications for combination therapy, and key boundary conditions related to hepatic and renal function. The rules were injected into the structured prompt as explicit constraint context to guide antibiotic selection and dose recommendation. The representative rules are provided in Table S2 in Multimedia Appendix 1.

We integrated three sources of information into a unified hybrid context: (1) patient context, including admission narratives and key laboratory indicators; (2) experience context, including the top 5 similar cases and their antibiotic regimens; and (3) constraint context, including clinician-defined rule constraints and guideline or knowledge graph evidence retrieved using LightRAG.

During inference, the hybrid context was organized into a structured prompt and submitted to LightRAG in hybrid mode. All model inference was conducted in Chinese. The comprehensive prompt template, including the detailed instructions for context integration and reasoning steps, is provided in Multimedia Appendix 1. Guided by key entities and cues in the patient context, the system retrieved local entity-level details and global relational summaries related to antibiotic selection, contraindications, and hepatic or renal dose adjustment. These retrieved results were then combined with the patient context and rule constraints for downstream reasoning. On the basis of this integrated context, the model generated antibiotic and dose recommendations through a stepwise reasoning process. The final JSON array was extracted from each model response and parsed using Python’s json.loads() function (Python Software Foundation). If parsing failed or the output was empty, generation was repeated up to 5 times. No manual correction was performed.

We implemented the proposed pipeline using DeepSeek-V3, GLM-4.6, and GPT-4o. Detailed model implementation settings are provided in Multimedia Appendix 1. To examine the contribution of each module, we evaluated four categories of experimental settings: (1) a base model without retrieval augmentation or clinician-defined rule constraints; (2) single-module variants, in which only one component was added; (3) dual-module variants, in which 2 components were combined; and (4) the full pipeline, which integrated similar-case vector retrieval, guideline-based knowledge graph retrieval, and clinician-defined rule constraints.

The framework was evaluated on both the internal test set and the external validation cohort. Two tasks were assessed: antibiotic selection and joint antibiotic selection plus dosing recommendation. For the antibiotic selection task, only the standardized medication name was considered during matching. For the antibiotic selection plus dosing task, medication items were matched after standardizing drug name, dose, dosing frequency, and route of administration. Dose values were unit normalized when possible, and a 5% numeric tolerance was allowed to avoid penalizing minor unit conversion or formatting differences. Performance was measured using Jaccard accuracy, precision, recall, and F₁-score. All metrics were calculated at the patient level based on true-positive, false-positive, and false-negative medication items and then averaged across cases. We used patient-level bootstrap resampling with 10,000 iterations to calculate 95% CIs for the performance metrics. To further assess robustness and clinical applicability, we conducted subgroup analyses in high-risk populations, including older patients and patients with hepatic or renal impairment. The detailed subgroup classification criteria are provided in Table S3 in Multimedia Appendix 1. Cross-site generalizability was examined using the external validation cohort.

This multicenter retrospective study was approved by the institutional ethics committees of Beijing Tsinghua Changgung Hospital (approval 23694-4-01) and Beijing Friendship Hospital (approval 2023-P2-031-01). The requirement for additional informed consent was waived by the ethics committees because this study used retrospective, deidentified clinical data and involved no direct patient contact or intervention. All extracted records were deidentified before analysis, and direct identifiers such as names, medical record numbers, phone numbers, and addresses were removed. The analytic data were stored in a secure research environment accessible only to authorized study personnel. No participants received compensation because this was a retrospective secondary analysis. No identifiable patient information appears in the manuscript figures, tables, or supplementary materials.

A total of 331 hospitalized patients with pneumonia were included in this study divided into a development cohort (n=233, 70.4% from Beijing Tsinghua Changgung Hospital) and an external validation cohort (n=98, 29.6% from Beijing Friendship Hospital). Baseline demographic characteristics, hepatic or renal impairment, and antibiotic class distributions are shown in Table 1. Compared with the external validation cohort, the development cohort included older patients and a higher proportion of male patients and patients with hepatic or renal impairment. The internal test set was broadly comparable to the overall development cohort in demographic and clinical characteristics, particularly in age and the prevalence of hepatic or renal dysfunction, although some variation was observed in antibiotic class distributions. Quinolones were the most commonly used antibiotic class in the development cohort, internal test set, and external validation cohort.

We compared different LLMs and integration strategies across 2 tasks: antibiotic selection and joint antibiotic selection plus dosing recommendation (Table 2). On the internal test set, the full pipeline with DeepSeek-V3 achieved the best performance among all evaluated LLM-based methods, with an F₁-score of 0.8110 (95% CI 0.7371-0.8762) and a Jaccard accuracy of 0.7624 (95% CI 0.6810-0.8386) for antibiotic selection and an F₁-score of 0.7538 (95% CI 0.6671-0.8329) and a Jaccard accuracy of 0.7076 (95% CI 0.6145-0.7938) for joint antibiotic selection plus dosing recommendation. The full pipeline with GLM-4.6 and GPT-4o also outperformed their corresponding base models but remained inferior to the DeepSeek-V3–based implementation.

On the external validation cohort, the full pipeline with DeepSeek-V3 maintained a strong performance (Table 3), achieving an F₁ score of 0.8605 (95% CI 0.7891-0.9252) and a Jaccard accuracy of 0.8571 (95% CI 0.7857-0.9184) for antibiotic selection and an F₁-score of 0.8503 (95% CI 0.7789-0.9150) and a Jaccard accuracy of 0.8469 (95% CI 0.7755-0.9133) for joint antibiotic selection plus dosing recommendation. In contrast, the base DeepSeek-V3 model without retrieval augmentation or clinician-defined rule constraints showed marked performance degradation, with an F₁ score of 0.4163 (95% CI 0.3289-0.5031) and a Jaccard accuracy of 0.3801 (95% CI 0.2951-0.4660) for antibiotic selection and an F₁ score of 0.3129 (95% CI 0.2303-0.3952) and a Jaccard accuracy of 0.2832 (95% CI 0.2049-0.3622) for joint antibiotic selection plus dosing recommendation.

To assess the contribution of individual modules, we performed an ablation study using the full pipeline as the reference configuration. The configurations of all ablation variants are summarized in Table 4, and the corresponding performance results are presented in Table 5. In the antibiotic selection task, the full model achieved an F₁-score of 0.8110. Removing clinician-defined rule constraints reduced the F₁-score to 0.7429, removing similar-case vector retrieval reduced it to 0.7156, and removing guideline-based knowledge graph retrieval reduced it to 0.7843. A similar pattern was observed for joint antibiotic selection plus dosing recommendation, where the full model achieved an F₁-score of 0.7538 compared with 0.6474 after removing rule constraints, 0.5946 after removing vector retrieval, and 0.7119 after removing guideline-based knowledge graph retrieval.

Performance declined further when multiple components were removed simultaneously. For example, the variant without guideline-based knowledge graph retrieval plus rule constraints achieved an F₁-score of 0.6253 (95% CI 0.5458-0.7012) for antibiotic selection and 0.5663 (95% CI 0.4829-0.6486) for joint antibiotic selection plus dosing recommendation, whereas the variant without vector retrieval plus rule constraints achieved F₁-scores of 0.6447 (95% CI 0.5589-0.7267) and 0.5641 (95% CI 0.4770-0.6529), respectively. The base model without retrieval augmentation or rule constraints showed the lowest overall performance.

We further evaluated model robustness for the joint antibiotic selection plus dosing recommendation task in clinically important subgroups defined by hepatic or renal function and age using the internal test set (n=70; Figure 2). In the hepatic or renal function subgroups, the full pipeline using DeepSeek-V3 achieved mean F₁-scores of 0.7062 (95% CI 0.5823-0.8198) in patients with impairment and 0.7939 (95% CI 0.6798-0.8991) in those without impairment. In the age-based subgroups, the corresponding mean F₁-scores were 0.7291 (95% CI 0.6177-0.8326) in patients aged 65 years or older and 0.8043 (95% CI 0.6884-0.9058) in those younger than 65 years. Overall, the integration strategy maintained relatively stable performance across all subgroups, whereas the base LLM approach consistently underperformed across the board.

To further assess recommendation safety and reasoning quality, respiratory physicians reviewed 45 error cases and a random sample of 50 generated recommendations. Error cases were categorized as safe but suboptimal or potentially unsafe. Of the 45 reviewed error cases, 43 (95.6%) were classified as safe but suboptimal recommendations, whereas only 2 (4.4%) were considered potentially unsafe. Both unsafe cases were related to insufficient consideration of hepatic or renal function during antibiotic selection or dose recommendation.

For reasoning quality evaluation, physicians assessed 50 cases across 4 dimensions: evidence relevance, rule applicability, reasoning consistency, and hallucination risk. For evidence relevance, 88% (n=44) of the cases received the highest score, and 12% (n=6) of the cases received a partial relevance score, with no cases rated as irrelevant. All reviewed cases received acceptable ratings for rule applicability, reasoning consistency, and hallucination risk, with no cases judged to contain incorrect rule application, inconsistent reasoning, or clinically significant hallucinations.

In this multicenter retrospective study, we developed and validated a constrained LLM-based CDS pipeline for antibiotic selection and dose recommendation in hospitalized patients with pneumonia. The proposed framework, which integrates dual-branch retrieval, clinician-defined rule constraints, and hybrid-context reasoning, consistently outperformed the corresponding base LLMs across both the internal test set and the external validation cohort. It also maintained a relatively stable performance in clinically important high-risk subgroups, including older patients and patients with hepatic or renal impairment. Notably, the higher performance in the external validation cohort may be partly related to differences in patient characteristics and prescribing patterns between the 2 hospitals. The external validation cohort had a more concentrated antibiotic distribution and relatively simpler medication combinations, whereas the development cohort included a broader range of antibiotic categories and more complex regimens, which may have made the recommendation task more challenging. Together, these findings suggest that combining external evidence retrieval with explicit clinical constraints can improve the consistency and robustness and provide preliminary evidence of cross-site generalizability of LLM-assisted antimicrobial decision support.

The performance gains were attributable to the complementary contributions of multiple components rather than any single module. Ablation analyses showed consistent performance declines after removal of clinician-defined rule constraints, similar-case vector retrieval, or guideline-based knowledge graph retrieval, with larger reductions observed when multiple components were removed simultaneously (Table 5). These findings indicate that the 3 information sources served distinct but complementary functions: similar-case retrieval provided patient-aligned experience-based references; guideline-based knowledge graph retrieval contributed structured evidence-based knowledge; and clinician-defined rule constraints encoded explicit boundary conditions for dose adjustment, contraindications, and hepatic or renal function. In inpatient pneumonia management, where antibiotic decisions are both time sensitive and high risk, parametric LLM knowledge alone may be insufficient. By integrating patient-specific context, external evidence, and explicit clinical constraints within a unified reasoning framework, the proposed framework improved the consistency and safety of antibiotic recommendations. This design is also conceptually aligned with the early empirical decision-making process typically performed during infectious disease consultation while remaining intended as a clinician support tool rather than a replacement for specialist consultation or clinician judgment.

Compared with traditional black-box prediction models, the proposed framework emphasizes a more interpretable, traceable, and auditable form of CDS. The system is not intended to replace clinician judgment but to function as an assistive tool for antimicrobial stewardship and medication review in hospitalized patients with pneumonia. Its practical value lies in its ability to rapidly integrate admission narratives, key hepatic and renal function indicators, similar-case experience, and guideline-based evidence to generate candidate recommendations for antibiotic selection and dose recommendation. At the same time, the system provides traceable evidence and rule trigger information, which may facilitate rapid clinician verification and improve the transparency of the decision-making process. Given that the external validation cohort differed from the development cohort in age, sex distribution, hepatic or renal impairment, and antibiotic class distribution, whereas the full pipeline still maintained strong performance, the findings further support the potential applicability of this approach in heterogeneous real-world settings.

This study has several limitations. First, the sample size was relatively limited, and the data were derived from only 2 hospitals. In particular, the external validation cohort included only 98 patients, and the distribution of antibiotic categories was imbalanced. This limits the strength of conclusions regarding cross-site generalizability. Therefore, further validation in larger, multi-regional, multilevel, and more category-balanced health care settings is needed. Second, although the reference labels were adjudicated by experienced physicians, the source information included real-world antibiotic orders, which are influenced by clinician experience, local prescribing preferences, and institutional antimicrobial stewardship policies. Accordingly, these labels should not be interpreted as an absolute gold standard, and such label-level heterogeneity may affect model generalizability. In addition, the drug library in the prompt was sorted by common clinical use, which may have introduced positional or frequency-related bias; future evaluations should test alternative drug list orderings. Third, stable structured body weight and BMI information was not available in the retrospective dataset. Consequently, the framework could not fully evaluate individualized weight-based dosing strategies for antimicrobial agents that require body weight adjustment. Future studies should incorporate structured body weight information to support more precise dose recommendation. Fourth, antibiotic options, resistance patterns, and clinical guidelines continue to evolve over time, so the knowledge graph, rule base, and retrieval resources will require ongoing maintenance and updating. In addition, the guideline-based knowledge graph was constructed from publicly available guidelines; therefore, we cannot fully exclude the possibility that some guideline knowledge was already present in the evaluated LLMs’ parametric memory. Future work should further evaluate the framework using recently updated guidelines or institution-specific antimicrobial stewardship pathways to better isolate the contribution of retrieval augmentation. Fifth, there was a temporal mismatch between the development and external validation cohorts. Although patients with confirmed COVID-19 pneumonia were excluded, residual differences in pneumonia etiology and antimicrobial prescribing behavior across time may still affect cross-cohort comparability. Finally, this study was retrospective and did not include prospective evaluation in real clinical workflows. Future work should assess the framework in human-in-the-loop practice settings and further examine its effects on prescribing quality, antimicrobial stewardship, and clinical outcomes.

In conclusion, we developed and externally validated a constrained LLM-based CDS pipeline for antibiotic selection and dose recommendation in hospitalized patients with pneumonia. By integrating dual-branch retrieval, clinician-defined rule constraints, and hybrid-context reasoning, the proposed framework improved the consistency and interpretability of LLM-assisted antimicrobial decision support and provided preliminary evidence of cross-site generalizability. The system also provided traceable evidence and rule trigger information to support clinician verification, highlighting its potential value for antimicrobial stewardship and medication review. Further prospective evaluation is needed to assess its performance and utility in real-world clinical workflows.