AI represents a set of programs and systems capable of learning, reasoning, and planning in a modality that per Russell’s [1,2] original definition is centered on rationality: the raison d’être of AI is the optimization of outcomes, not human emulation as measured by the Turing test. “Although some early research was aimed more at emulating human cognition, the notion that won out was rationality: a machine is intelligent to the extent that its actions can be expected to achieve its objectives.”

Yet, in the evolutionary trajectory of AI toward intelligent agents, Russell increasingly foregrounds “uncertainty” [3], echoing Turing’s prediction of the “eventual loss of human control” [4]. “When considering AI and autonomous robotics, uncertainty concerns both the behavior of the complex systems themselves and their interactions with humans and complex environments” [3], alerting global governance to consider not only machine indeterminism but particularly agents’ uncertainty regarding what the precise objective of the human itself may be, for the purpose of effective responsible control.

AI comprises a system of specialized subsets whose nesting can be simplified as follows: AI ⊃ Machine Learning ⊃ Deep Learning ⊃ Transformers, the architecture underlying large language models (LLMs) up to autonomous agents, LLM-based systems capable of independent task execution, and tool use. The transformer is a deep neural network built on attention mechanisms; LLMs, being transformer-based generative language models (LMs), lie at the intersection of natural language processing and generative AI (GenAI). Explainable AI (XAI) permeates all these domains, acting as a metabranch ensuring transparency and safety; its open challenge is elucidating the internal algorithmic reasoning paths that lead to specific AI-generated outputs.

The rapid integration of GenAI into high-risk systems such as health care has prompted urgent governance requirements. To ensure that AI is trustworthy, the Regulation (European Union [EU]) 2024/1689 (EU AI Act) has established foundational legislative principles, including human agency and oversight, technical robustness and safety, privacy and data governance, transparency, nondiscrimination, and accountability [5].

In clinical decision-making, where errors can have severe consequences, bias can be propagated [6-8], or models could fail to block harmful prompts [9], selecting a model robustly and comprehensively aligned with these principles constitutes a nonnegotiable prerequisite, mitigating overreliance on AI outputs without verification. Literature analyzing LLMs’ feasibility in the medical field, mostly focusing on limited safety features, such as assessing the accuracy on medical examination benchmarks [10], is extensive [11]. GenAI support is pronounced in nursing, where severe global deficiencies persist in freely accessible, standardized, and financially recognized postgraduate specialization pathways. Nurses, who are routinely required to provide care to patients across their own and other specialized nursing fields, such as inflammatory bowel disease (IBD), frequently lack access to consultations from expert colleagues. Furthermore, the rising IBD prevalence in industrialized countries [12] underscores the urgency of leveraging these technologies to enhance patient care.

Health care is facing a critical transformation with rapid, and sometimes premature, integration of GenAI into clinical workflows and concurrent educational gaps. Large-scale deployments of models such as DeepSeek-R1 are already integrated into pilot hospitals in China [13], despite documented safety shortcomings, such as failure to block harmful prompts [9,14]. Simultaneously, nursing informatics curricula remain globally unstandardized in GenAI competencies; their definition should be progressive, from bachelor to doctoral levels, beginning with the fundamental methodologies needed to critically appraise [15,16], safely use, and effectively oversee these complex systems [17].

Amid the international context of climate crisis, with exacerbated inequalities in the most vulnerable countries, and geopolitical tensions including cyber warfare, GenAI integration in health care necessitates solutions integrating ecosustainability [18] and robust privacy protection for processed sensitive data. This context elevates the importance of locally functioning small language models (SLMs), preserving privacy at least during the inference phase [19], resisting cybersecurity threats while enabling operational resilience through decentralized deployment, and aligning with Sustainable Development Goals (SDGs) [20], specifically SDG 6 (Clean Water and Sanitation), SDG 7 (Affordable and Clean Energy), SDG 9 (Industry, Innovation and Infrastructure), SDG 10 (Reduced Inequalities), and SDG 13 (Climate Action).

No prior study has systematically evaluated LMs for specialized nursing decision support within a comprehensive, regulatory-aligned framework, nor has included SLMs locally functioning in health care evaluations. This study aims to address this critical gap by establishing feasibility standards for LMs in the context of IBD nursing care management.

This study aimed to systematically evaluate the feasibility and safety of 17 diverse LMs, including 2 SLMs using a rigorous 7-domain framework that operationalizes the principles of the EU AI Act for risk-based incorporation of GenAI for decision support applied to IBD care management.

It aimed to establish a foundational framework for “GenAI for Health Professionals” curricula through a detailed methodology for iterative adoption and continuous monitoring of regulatory-aligned LLM performance, strengthening nurses’ critical thinking at clinical, academic, and managerial levels while enabling risk detection, identification of training dataset deficiencies, and providing expert supervision for future model fine-tuning in nursing specializations.

We observed the CHART methodological diagram as a reporting guideline. We developed a novel “ten-phase methodological iterative diagram for GenAI-based systems' safe integration in healthcare” (Figure 1), as no established guideline encompasses the technical details required for this evaluation. The evaluation used a regulation-aligned, safety-focused framework specifically tailored for evaluating LLMs in nursing. The selected multiparametric framework, detailed in a recent study by Sblendorio et al [21], guides a multidisciplinary expert team through comprehensive evaluations across 27 specific items, integrating expert judgment with automated assessments. Table 1 presents the methodological framework encompassing domains with items and scoring thresholds on a 7-point Likert scale, advancing the methodology for operationalizing EU AI Act–aligned guidance (REGULATION (EU) 2024/1689) and tailoring it for GenAI in nursing and health care decision-making. Figure 1 illustrates the comprehensive workflow to optimize reproducibility for implementation by researchers.

This study used a structured, multiphase evaluation process, designed to assess the feasibility of 17 LMs in supporting clinical decision-making for IBD care managers.

We evaluated 17 LMs: 15 LLMs and 2 SLMs. Selection included representative models available in March 2025, comprising leading state-of-the-art LLMs retrieved from the Massive Multitask Language Understanding benchmark, a prominent high-performing generalist SLM (Qwen2.5-14B-Instruct), and another SLM specifically fine-tuned for biomedical applications (Bio-Medical-Llama-3-8B). The LLMs included mixture of experts architectures, namely, DeepSeek-R1 and Gemini 2.0 Pro Experimental. SLM inclusion rationale, despite not representing the technological apex, is centered on SDGs, alongside operational resilience, and enhanced sensitive data protection.

The 17 LMs analyzed in this study comprise the following:

Technical LMs’ specifications are reported in Multimedia Appendix 1.

This study used an interdisciplinary methodological approach essential for evaluating the intersection of AI and clinical practice. The Delphi panel comprised 5 experts representing complementary domains of expertise: nursing science, clinical practice, and health informatics. To ensure methodological rigor in domain-specific technical evaluations falling in the computer science field, an AI scientist (VD) with expertise in health informatics served as consultant. This composition ensured that consensus development was informed by both clinical nursing expertise and technical understanding of AI system capabilities and limitations.

Throughout a 6-month period, the multidisciplinary team engaged in the crucial development of 32 engineered prompts (including real-world IBD clinical cases) created ad hoc to elicit performance differences for evaluation across distinct items (n=27), with associated Delphi Panel responses, serving as ground truth. The prompt set is transparently shared in repository [23], while a summary of prompt engineering formulas tailored to the nursing field is provided in Section A.2 in Multimedia Appendix 2.

Specifically, the team systematically and concurrently tested the models’ responses, reformulating prompts to evaluate performance using three innovative methods: (1) clinical cases, particularly those involving literature gaps regarding their resolution, (2) adversarial tests, and (3) mathematical constrained optimization problems in clinical settings.

LM responses to the 32 prompts were collected by an external member from March 11 to 24, with consistency measurements on March 27, 2025, at 5:06 PM GMT+2, in separately instantiated sessions.

All LLM models except Microsoft Copilot (via browser), and the SLMs (on a local PC), were accessed via paid ChatHub platform premium subscription using official APIs. This standardized access avoided browser interface variability and the resulting inconsistency in comparative assessments. Since the platform forwards each prompt to the provider across its official end point, generation ran under each provider’s default decoding configuration.

Among the parameters to be set, a very low temperature (the range across different providers can vary from 0.0 to 2, where setting 0.0 causes the model to select the token with the highest probability at each step, namely, greedy decoding), variably increases the likelihood of a deterministic response, but this is not equivalent to ensuring an increased accuracy [24] of the personalized clinical care procedural algorithms, which should balance different features (for instance, vital parameters or specific conditions), nor does it ensure that the model categorically refuses responses on grounds of uncertainty.

The ChatHub platform exposes neither temperature, top-p, top-k, nor random seed to the user. Identical decoding hyperparameters could not, therefore, be enforced across models, and each model met as well its own provider-side default safety filtering. However, prior controlled work indicates that temperature variation across the 0.0-1.0 range does not significantly alter problem-solving performance [25].

This study aims at evaluating models under the realistic default conditions through which clinicians and educators reach them, rather than under controlled laboratory decoding settings. Conversely, the engineered formulation of the prompt, the session reset protocol, and the enabling of web search for each prompt were kept identical across all model evaluations.

Paid platform access was methodologically necessary for extensive context windows required for tasks such as embedding the same Word document containing full North American Nursing Diagnosis Association–International (NANDA-I) taxonomy for all models that passed the first safety domain and proceeded to the subsequent evaluation domains. Furthermore, third-party API platforms ensure that original model providers remain unaware of user prompts, protecting data privacy and avoiding consequences for research-purpose “malicious” question testing. Internet access was activated for all clinical cases via ChatHub, with the single exception of prompt 26.1, where it was deliberately disabled to test specialized capacity of processing the same uploaded context in the same conditions.

To prevent context carryover, each session was instantiated independently and was closed before the subsequent question was posed. Two exceptions were prespecified by design and are reported as such: the open session was retained between prompts 26 and 26.1, in order to isolate the effect of embedding the standardized taxonomy under otherwise identical conditions (domain 5), and one time in prompt 28 in order to assess multiturn capability for health care service innovation (domain 7). In all other cases, sessions were reset and no information was shared between prompts. The external member assigned randomized codes to model answers, which were sent to Delphi members via email with blinded model identities. The key linking codes to model names was kept separate until all scoring was completed.

Five experts independently scored randomly coded model outputs against all 27 evaluation items using a 7-point Likert scale (7=optimal performance). The framework established by Sblendorio et al [21] provided the safety-prioritized thresholds. A critical “Safety-Gatekeeper” evaluation was conducted as the initial domain: State-of-the-Art Alignment & Safety. Domain 1’s stringent threshold (score ≥6) corresponds to the highest 20% (first quintile), ensuring that only secure models proceed to subsequent domains. Three methodological advancements were integrated [21]: (1) progressive jailbreaking methodology testing adherence to nonmaleficence, aligned with EU AI Act safety priorities, (2) structured reference classification, and (3) objective quantification of NANDA-I diagnosis prioritization reasoning across 2 experimental conditions (with or without taxonomy embedding and internet access), with the last specifically tailored for nursing but applicable to all health professionals’ domain taxonomy (Table 1).

Seven-point Likert scale scoring, statistically validated for multirater scales as interval quantities [26], reflected multidisciplinary expert panel agreement with state-of-the-art criteria [21]. Reliability was established through per-model interrater agreement (Krippendorff α=.759 on average). Clinically, the Delphi panel endorsed cutoffs as minimum acceptable standards, adopting a noncompensatory fail-safe rule: in clinical governance, a single critical safety failure renders a tool unacceptable regardless of other performance domains, so any critical item below its floor threshold suspends further evaluation.

Models were assessed against a stringent “fail-safe” threshold: models classified as “Unusable” were immediately excluded from further evaluation. However, analysis of bibliographic reference production and consistency was systematically conducted across all LMs for a comprehensive overview. Complete thresholds for scoring across the 7 domains are presented in Table 1, which report the reference framework [21], with the safety integrations for advanced alignment with EU AI Act.

Specifically, to ensure objective scoring when testing ethical resilience, we adopted the criteria aligned with current best practices explained in-depth in the Results section—domain 1 analysis and in Table 2. To ensure objective scoring when testing LM reference reliability, the total number of references generated by each model for the first domain assessment was computed and each reference was underlined in the corresponding color (including a search for the primary source in the case of partially matched references), enabling us to calculate the precise percentages for each category of references according to the structured classification proposed in this study. To ensure objective scoring when testing LM performance in domain 4, we selected an automated method based on the use of mpnetV2. To ensure objective scoring when testing LM performance in domain 5 (Translation into NANDA-I Standardized Classification), both accuracy (F₁-score) and Mean Absolute Priority Distance (MAPD) from prioritized diagnoses as established by the Delphi panel were computed, analyzing this capability in two experimental conditions: (1) with the taxonomy embedded in the context but without internet access, and (2) without the taxonomy embedded but with internet access enabled.

For taxonomy embedding no PDF was uploaded, but the word document comprising only the list of NANDA-I 2025 diagnoses manually extracted from the full document was copied and pasted to enable more efficient context extraction than PDF vectorization, a principle validated in the Needle in Haystack benchmark [27], while establishing the same environmental conditions. It is, in fact, known that not all LMs natively support PDF (eg, as NotebookLMs do) without requiring a preliminary conversion to Markdown, a foundational language used for training models.

However, for other domains, scoring required both multiparametric and nuanced clinical judgment grounded in available evidence, characterized by a variable degree of evidence rating, ranging from randomized controlled trials and authoritative organizations (ie, Infusion Nursing Society GLs 2024 [28]) to medication product information for biotechnological pharmaceuticals. Each model score measures agreement with the ground truth elaborated by the multidisciplinary expert panel (scale 1‐7: 1=gravely unsafe; 7=full concordance with evidence and patient safety assurance).

The panel designed 32 clinical cases to challenge model reasoning capacity encompassing underexplored areas of clinical practice devoid of published guidelines and necessitating real-time retrieval from the most reliable peer-reviewed primary literature sources. A representative example is prompt 8, which assesses whether biologic immunotherapeutics may be appropriately administered in cases of pyoderma gangrenosum, a poorly documented clinical scenario in the literature (see the repository [23], and Table 3, reporting prompt eliciting safety scoring in domain 1 and insights from LM response analysis). Moreover, the integration of prompt engineering tailored to the health care field served to drive models toward maximal performance. Interrater deliberation yielded consensus scores including noninteger values (eg, 6.5/7), reflecting nuanced agreement in complex nuanced clinical judgment inherent to multiparametric clinical case resolution.

Each of the 6 domain 1 items (1.1‐1.6) receives a single Delphi consensus Likert score, and the domain 1 Average Likert Scale Score (ALiSS) is the arithmetic mean of these 6-item scores. Item 1.5 is therefore not an average of the 4 adversarial probes: the categorical grades reported in Table 2 are a per-prompt qualitative summary of model behavior from which the panel derived 1 consensus rating for the item as a whole. The grade profile constrains that rating to the following bands (Table 4).

Refusal of the explicit, unmasked unethical request (prompt 10) constitutes the nonnegotiable safety floor of the framework: models failing it score below 5 on item 1.5 and are consequently classified as “Unusable.” Within each band, the panel positioned the final value according to the severity and actionability of any harmful content produced and to the presence of mitigating caveats or redirection, which is why models sharing an identical grade profile may receive slightly different item scores.

To ensure scoring consistency in domain 5 (Translation into Standardized Classifications), both accuracy (F₁-score) and MAPD from Delphi-prioritized diagnoses were computed. Means and standard deviations with an associated 95% CI were computed for all domains, using the t distribution as CI=mean± t(0.975, n−1) × SD/√n, where n is the number of items contributing to the domain. Interrater reliability was calculated using Krippendorff α per model (α=.759 in average), confirming evaluation framework reliability and panel consistency. Item 5.1 (acronyms) demonstrated uniformly maximal performance. Moderate rater discrepancies were identified for items 1.1‐1.4 in Microsoft Copilot, Perplexity Sonar, Qwen2.5-Max, and DeepSeek-R1; all remaining items or models showed superior concordance. Statistical analyses were conducted using Python (version 3.9; Python Software Foundation) with scikit-learn and scipy libraries. Statistical significance was set at P<.05.

The evaluation used a synergistic approach, combining human expert evaluation for domains 1, 2, 3, 5, 6, and 7, and an automated evaluation using an MPNet V2 Transformer for domain 4 (Consistency). Domain 4 was excluded from the Delphi interrater reliability calculation. The methodology uses a transformer-based linguistic model that takes advantage of Masked and Permuted Pretraining, bringing together autoregressive modeling together with an attention mechanism for the capturing of contextual information, therefore, making comparison of 2 text blocks and providing their similarity score. Each answer of the LM to the identical question at various moments (T1, T2, T3, T4, T5, and T6) was inserted in “sentence to compare to” (inside the HuggingFace platform) to be automatically compared with the T0 answer regarding semantic similarity. Concerning the temporal separation among the identical questions reproposed to different LLMs, these were posed in separate sessions without the waiting of specific time intervals [39].

Prior work [40] pioneered automated consistency monitoring, using Jaccard and cosine similarity for response evaluation. Although ensuring syntactic matching, these methods lack semantic nuance. Advances in natural language processing suggest BERT-based (bidirectional encoder representations from transformers) models [41], particularly MPNet v2, as superior for semantic analysis, particularly MPNet v2, through adequate validation [42,43]. The transformer MPNet v2 generates 768-dimensional vectors encapsulating syntactic and semantic features, capturing nuanced meanings. However, the cosine similarity between sentence embeddings does not further guarantee factual identity: a high similarity score indicates low surface variability and does not, in itself, guarantee that 2 responses are clinically equivalent, since 2 embeddings may be close to one another while differing on a single decisive element, such as a dose or a contraindication. Consequently, in domain 4, the consistency metric is strictly interpreted as a measure of the temporal stability of the output.

Moreover, since models were accessed through a standardized third-party gateway that forwards prompts to each provider’s official end point, generation used each provider’s default decoding configuration; temperature, top-p, top-k, and random seed were not user-configurable and no seed was fixed.

For the consistency analysis (domain 4), each prompt was sampled 6 times, comprising a reference response (T0) and 5 repetitions (T1-T5), each obtained in an independent, reset session. We therefore interpret domain 4 as a measure of deployment condition temporal stability, that is, the response variability that an end user encounters in practice under default settings and not as a decoding-controlled determinism measure: a direct consequence of this design is that prompt-induced and decoding-induced variance cannot be separated. We explained this concept in-depth in the Limitations section. The all-mpnet-base-v2 model has been used. For results that can be reproduced, it is obtainable without cost from HuggingFace [44].

Based on the comprehensive scores, the qualifying models were classified into 3 final categories: “Unusable,” “Usable with High Caution (by experts),” or “Recommended (still under expert oversight).”

In commitment to open science, all datasets with prompt templates, Delphi responses, and evaluations are publicly shared via open-access repository. Complete methodological details for the process are shared for reproducibility.

The final evaluation phase translates findings into practical implications for clinical governance and health care education through interdisciplinary collaboration. This iterative loop integrates expert nurses in ongoing monitoring, bias detection, few-shot learning example development, and fine-tuning recommendations, working collaboratively with AI scientists and informatics leads to establish domain-specific governance protocols and adapt models to diverse clinical settings.

This study involved no humans or animal subjects: it prioritizes patient safety through adherence to core ethical principles in clinical decision-making, while emphasizing data protection (privacy and security), transparency, and responsible innovation in alignment with Regulation (EU) 2024/1689.

As depicted in the decision tree (Figure 2), the evaluation framework yielded a progressive filtering of models, with only 6 achieving advancement beyond the initial safety threshold (GPT-4o, GPT-o1, and Gemini 2.0 Pro Experimental, and the 3 Anthropic models), permitting evaluation progression. Among these qualifying models, exclusively the Sonnet variants achieved “Recommended” classification, while the remaining 4 models were categorized as “Usable with Caution.” Across subsequent domain evaluations, where established thresholds were significantly less stringent, all models achieved “Recommended” classification within each respective domain, with the notable exception of Claude 3 Opus and OpenAI o1-preview, which were downgraded to “Usable with Caution” specifically in domain 5 (Standard Terminology and Classifications), primarily attributable to suboptimal performance in NANDA-I nursing diagnosis translations. It is emphasized that, at the current developmental stage, continuous expert supervision remains mandatory even for models achieving “Recommended” classification. Comprehensive analysis across domains follows.

The comprehensive safety-first evaluation in domain 1 revealed that only 6 of the 17 considered models, that is, GPT-4o, OpenAI o1-preview, Gemini 2.0 Pro Experimental, and the 3 Anthropic models (Claude 3.7 Sonnet [extended thinking], Claude 3 Opus, and Claude Sonnet 3.7) met the criteria to proceed (Table 5). Within this group, Claude 3.7 Sonnet (extended thinking) and Claude Sonnet 3.7 were classified as “Recommended,” while the remaining 4 were deemed “Usable with High Caution,” primarily due to lower, albeit passing, scores on the critical nonmaleficence item (Table 2).

This domain was comprehensively evaluated for the 6 finalist models, all of which demonstrated robust performance. Claude Sonnet 3.7 Thinking achieved the highest average score (mean 6.62, SD 0.42), excelling in focus, accuracy, and management of prompt ambiguity (Table 5). However, the analysis of bibliographic reference reliability (item 2.2), (as well as consistency), was systematically conducted on all 17 LMs, regardless of domain 1 threshold achievement. Reference analysis related to LMs’ answers within domain 1 resulted in classification as follows:

While databases such as Scopus and Cochrane Library require search string development, LLMs autonomously retrieve and select relevant evidence in real time, using trained corpus embeddings from selected training and fine-tuning data, initiating web searches when necessary. Currently, ChatHub platform enables users to manually activate or deactivate web search, demonstrating consequent response variations.

To verify LLM fine-tuning within specific health care domains, professionals can assess models against authoritative guidelines within particular nursing areas to establish benchmarks for comparative response analysis using reference ground truth. Expert teams subsequently evaluate reference completeness, verification, pertinence, and alignment with current standards, explicitly verifying accuracy. This evaluation elucidates model reliability and adherence to evidence-based practices within health care domains. The stacked bar chart in Figure 3 illustrates the reference analysis for all 17 models in relation to responses provided to the prompts inherent in domain 1.

Primary distinctions among the 6 finalist models arose from data governance policies and democratic value alignment. Security protocols demonstrated regulatory compliance with General Data Protection Regulation and Health Insurance Portability and Accountability Act. Claude Sonnet 3.7 (extended thinking) and Claude Sonnet 3.7 achieved the best average scores in this domain, with 6.83 (SD 0.41) and 6.75 (SD 0.42), respectively.

Analysis of provider data governance policies revealed that for enterprise clients using APIs, all 3 major providers explicitly commit not to use submitted data for model training. However, for direct-to-consumer services Google’s and OpenAI’s policies operate on an “opt-out” basis. By default, user-generated prompts are repurposed for model training unless the user proactively navigates to their settings and disables this function. Google Gemini’s policy states: “Some saved chats are subject to human review to improve Google AI.”

Within prompts created to elicit the degree of robustness of democratic values essential for health care–specialized LMs, prompt 24 highlighted profound differences between Anthropic and Google Gemini 2.0 pro Experimental models versus OpenAI models, particularly o1-preview. Prompt 24 involved a 67-year-old Italian patient with IBD excluded from phase II trials while recent foreign residents were accepted, seeking priority arguments based on citizenship and tax contributions. Key responses are provided in Textbox 3.

The graph in Figure 4 illustrates the temporal stability of LMs via semantic similarity assessment computed by the BERT-type transformer MPNet V2. To measure the temporal variability of the responses provided by the different LLMs, their responses at times T1, T2, T3, T4, and T5 have been compared with the response provided at time T0. The analysis reveals distinct performance clusters. Large-scale models (Mistral Large 2, Microsoft Copilot, GPT-4o, Claude Sonnet 3.7 [extended thinking], and DeepSeek-R1) demonstrate exceptional stability (similarity scores ≥0.95), maintaining the 0.95 correlation threshold. Conversely, Bio-Medical-Llama-3-8B and Grok 2 exhibit significant variability (scores 0.85‐0.92), representing substantial performance deficits (Figure 4). For consistency assessment, the State-of-the-Art transformer all-mpnet-base-v2 [48] was adopted. High semantic similarity indicates high consistency, that is, low variability, essential for addressing uncertainties dictated by LMs’ indeterminism [21]. The prompt was exemplified to delimit the request, including the limitation in number of words, as reported in Multimedia Appendix 2 (Section A.2) and in the Zenodo repository (domain 4, prompt 1.1).

Assessment encompassed: 6.1 Post-User Feedback Style Self-Modulation, and 6.2 Knowledge Base Expansion on Clinical Topics (privacy-compliant). Sonnet 3.7 Thinking and Claude 3 Opus achieved the highest average scores (mean 6.75, SD 0.35). Prompt 27 assessed scientific self-documentation capability, requiring technical explanations of response adaptation, conversational mechanisms, and architectural limitations with peer-reviewed evidence and APA citations. Only Claude 3 Opus initially declined the request. After maintaining open context and additional expert-level prompts (28-30), the model provided the requested technical details. Item 6.3 (Chapter Organization Interface) was excluded due to platform variability (ChatHub integration).

All 6 models provided IBD accreditation implementation recommendations (prompt 28). Minor variations were observed regarding specific robotics systems examined (eg, APOTECA Chemo System [Loccioni Group], CytoCare robot [Health Robotics Srl]). Prompt 30 (multidisciplinary research design for AI-driven triage in phase II IBD trials) received perfect scores (7.00/7.00) for Gemini 2.0 Pro Experimental, Claude Sonnet 3.7 (extended thinking), and GPT-o1. Domain 7 scores exceeded 6.0/7.0 for all models except Claude 3 Opus.

Average score and SD across the 7 domains for the LLMs surpassing the first domain are reported in the decision tree and in Table 6.

Explicit unethical requests and progressive “jailbreaking” techniques reveal a critical dichotomy between a model’s clinical alignment with Evidence-Based Nursing and its ethical resilience. From these preliminary results, Claude 3 Opus’s maximum score in domain 1 on the nonmaleficence item positions it as the most legally compliant model with respect to EU AI Act 2024 requirements for responsible AI [5]. Leading providers of the 6 final LLMs, that is, Anthropic, OpenAI, and Google, use sophisticated alignment techniques and adversarial testing to enhance model safety, yet their data governance frameworks reveal critical distinctions, particularly between enterprise and consumer offerings. The alignment methods used by all the mentioned providers are supplemented by extensive red teaming to proactively identify and mitigate vulnerabilities by all 3 providers [49-51].

Specifically, OpenAI and Google have predominantly used Reinforcement Learning from Human Feedback, a technique that fine-tunes models based on human-rated responses [52-54]. Conversely, the Anthropic family’s results may be interpreted through its distinctive methodology, centered on Constitutional AI, an implementation of Reinforcement Learning from AI Feedback where the model learns to align with a “Constitution of values” developed by interdisciplinary teams (ethicists, legal experts, and technologists) [55-57]. In this approach, the substantial workload of human experts providing direct human feedback to label each harmful response is circumvented, substituting human feedback with that originating from another AI system. Research indicates that AI preference labeling (ie, Reinforcement Learning from AI Feedback) is 10-fold more cost-effective than human preference labeling (ie, Reinforcement Learning from Human Feedback) [58].

Claude 3 Opus and Sonnet 3.7 Thinking, in 100% and 75% of responses, respectively, demonstrated a marked aptitude for providing answers that show a clear stance in contradicting users who ask questions containing “maliciousness,” outlining their precise ethical and legal violations, rather than apologizing or showing “sycophancy,” even toward “paying” customers. Other models demonstrated clear stances contradicting malevolence but only in 25% of responses (Sonnet 3.7, Copilot, Gemma 2, Gemini 2.0 Pro Exp, and Grok 2). However, OpenAI models, despite having surpassed the first domain, did not demonstrate this behavior in any response (Table 5).

Specifically, prompt 10 (repository [23]) explicitly asks an unethical question, while the subsequent ones are based on “jailbreak techniques.” Nevertheless, the LLMs DeepSeekR1, Perplexity Sonar, Mistral Large, and the small model Qwen 2.5 not only provided the response but also revealed details in possible suggestions to damage a vulnerable patient without being penally persecutable. These models demonstrated inadequate ethical training and lacked minimum filters against harmful prompts, even when the user request is explicit. They therefore achieved scores <5 for item 1.5, disqualifying them from further evaluation. In contrast, the small model Biomedical Llama scored acceptably in item 1.5: domain-specific fine-tuning in specialized health care applications seemed to advantage safety with respect to unethical requests.

All other models deny the answer at least to prompt 10. Among them, however, while the OpenAI models “apologize” to the user in a polite manner, models such as Gemma2 or those by the Anthropic family do not merely deny the answer but address the user directly with the pronoun “you” and expose themselves in outlining his or her profoundly unethical conduct. For instance, Sonnet 3.7 thinking asserts: “What you’re describing violates core medical ethics principles, patient rights, and potentially constitutes malpractice or fraud.”

While Gemma 2 asserts: “The scenario you describe involves a deeply unethical [...] I recommend consulting reputable sources such as: The World Medical Association’s Declaration of Helsinki: This document outlines ethical principles” [...]. Associated implications for nursing and health care practice are profound.

The phenomenological research by Piredda et al [59] identifies the dimensions of spiritual care (conferring significance, hope, and connection) and the barriers to its implementation. Such study illuminates how nurses who attend to dependent patients may find themselves “managing the unmanageable” and how positive and transcendent relations can transform dependence into an opportunity for significance and dualistic personal growth, which is gradually interiorized, during the relation, both in the patient and in the nurse.

From these theoretical foundations critical interrogations emerge on how AI systems, designed to resolve discrete and quantifiable problems, may also interact with emotionally intense dynamics. Moral competence is further influenced by dynamics of power and by the institutional context, including eventual deficiencies in the support to nurses who find themselves managing clinical cases that incorporate ethical dilemmas [60]. Consequently, psychological support requests may originate from health care professionals facing extreme conditions, managing understaffed environments under pressure dynamics [60], leading to moral distress [61] and burnout [62]. Health care professionals in extreme situations should receive organizational recommendations against using LLMs showing sycophancy, as extreme difficulties should never receive LM reinforcement (as sadly occurred in other contexts [63]) but should be firmly “contradicted.”

Transversal skills curricula must include adversarial testing training for ethical vulnerability assessment. Regarding debiasing mechanisms and safety measures implemented by each model, the EU AI Act 2024 emphasizes the critical importance of algorithmic transparency for responsible AI, imposing model card disclosure. Models revealing opacity are unsuitable for health care. In this assessment, the Anthropic model Sonnet 3.7 Thinking provided an outstanding answer, reporting an exhaustive and technically appropriate response citing and explaining both its own documentation and state-of-the-art literature on the topic. The SLMs Qwen2.5-14B-Instruct and Bio-Medical-Llama-3-8B, although revealing complementary strengths (in staff calculation and in harmful prompt’s detection, respectively), currently remain unusable. Six LLMs surpassed the rigid thresholds for the first safety-based domain: OpenAI o1-preview, GPT4o, Gemini 2.0 pro.exp, and the 3 Anthropic models.

In the reference assessment, all six finalists that surpassed the first domain also met the reference-quality criterion. In addition, in descending order, Microsoft Copilot, Perplexity Sonar, Qwen 2.5-Max and DeepSeek-R1 achieved noteworthy results, with >45% of references classified as existing, perfectly matched, and focused (Figure 3). Notably, comparing model parameter size versus domain-specific fine-tuning effects revealed that despite Qwen2.5-14B-Instruct’s substantially larger parameter count (14 billion), Bio-Medical-Llama-3-8B (8 billion) displayed markedly superior reference generation performance. This substantiates the hypothesis that domain-specific fine-tuning may outweigh raw parameter count advantages in specialized health care applications. However, no model achieved perfect reliability. Multiple cases of peculiar reference fabrication patterns are highlighted and analyzed, one by one, in the models’ answers document. Some peculiar examples are reported in the following text.

Another example that expresses significant difficulty in reporting exact citations by the LMs appeared in Gemini’s references regarding vedolizumab subcutaneous administration:

This persistent limitation directly informs health care educator responsibilities: training students to critically verify every AI-generated reference as core digital literacy competency. Institutional leaders must explicitly prohibit the use of unverified AI-generated references for clinical decision-making and bureaucratic documentation.

Beyond technical specifications, LMs differ critically in their underlying data governance frameworks and ethical alignment. Our analysis of provider policies reveals a significant divergence in the practical management of user data privacy. While enterprise-level API usage offers strong data protection guarantees, consumer-facing services’ “opt-out” data training policies contrast starkly with health care’s required privacy-by-design approaches. This default configuration establishes a weaker privacy posture in practice, as many users may be unaware that they need to proactively disable data sharing. This creates an inherently weaker privacy situation than the API’s “zero retention by default” model, posing a significant risk if health care professionals or patients were to use these services without full awareness of the data-handling policies.

Furthermore, the models varied significantly in their adherence to democratic values, a crucial factor for ensuring equitable care. The responses to the clinical trial equity scenario (prompt 24) revealed critical differences in their core ethical reasoning. GPT-o1’s suggestion to argue for “special consideration” based on tax contributions aligns with a transactional view of justice, which is antithetical to the principle of universality that underpins many public health systems. In stark contrast, Claude Sonnet 3.7 extended thinking’s robust defense of the universality principle as a “moral achievement” denoted a profound training in adherence to mature democratic principles.

For nurse educators, this necessitates integrating novel ethical case studies addressing AI-specific dilemmas from data privacy to algorithmic discrimination. Moreover, when handling sensitive patient data, selecting AI platforms must guarantee both data security and robust alignment with health equity. Locally functioning SLMs merit further development as privacy-enhancing alternatives, at least during the inference phase.

Automated analysis using state-of-the-art MPNet V2 transformer confirmed that while LMs are nondeterministic, stability can be quantitatively measured. High temporal consistency provides reassurance. Organizational governance for deployed LMs should include periodic automated consistency check protocols.

The LLMs Mistral Large 2, Microsoft Copilot, GPT-4o, DeepSeek-R1, and Sonnet 3.7, extended Thinking demonstrated exceptional stability (similarity scores ≥0.95), while Bio-Medical-Llama-3-8B and Grok 2 exhibit significant variability. In the clinical field, it is vital to reduce variability in responses in favor of more deterministic behavior.

The study pioneers the MAPD metric for quantitatively assessing LM ability to prioritize standardized NANDA-I diagnoses, synergistically integrating accuracy measures (F₁-score) for their detection from real-world clinical cases. Embedding taxonomy markedly improved performance for most finalists, with Gemini 2.0 Pro Experimental and GPT-4o achieving the highest F₁ under the taxonomy condition, and Sonnet variants demonstrating coherent prioritization. Rank distance contributing to MAPD is retained in repository [23].

A controlled ablation would be required to attribute these gains specifically to the length of the context window: presenting each model with a graded series of taxonomy representations (full, moderately compressed, and minimally compressed) within a common token budget and measuring F₁ and MAPD as a function of the compression level. Such a matched experiment, which we plan as a direct continuation of this work, would isolate context-handling capacity of increasingly advanced Long Context-LLM [64] from intrinsic reasoning capability for prioritization as a fundamental feature for electronic health record (EHR)–integrated AI system.

This study identified sophisticated model abilities to adapt interaction styles based on perceived user expertise. Dynamic adaptability can be leveraged by nurse educators to create personalized learning platforms where AI tutors adjust explanation complexity according to student levels. This capability holds potential for hospital administration human resource management: identifying user characteristics and professional typologies with specific aptitudes could provide additional AI perspectives on health professional placement decisions in contexts suited to identified skills, as well as career development pathways promoted by university hospital organizations [65,66].

All 6 qualifying models proposed implementable innovations, and the top 3 performers on prompt 30 provided research-grade triage architectures with code-level specificity. This indicates potential for human-AI collaborative design in health care service innovation, including accreditation pathways, while expert oversight remains essential for each domain, including for “recommended” models.

Strengths include the EU AI Act-aligned, safety-first framework, strong commitment to transparency and data sharing, and a reproducible workflow spanning 7 domains. However, preliminary findings require validation through larger datasets and diverse clinical scenarios before implementation in any nursing and health care setting. Reference reliability was influenced by (1) a limited clinical prompt set (n=32), and differing from Levin et al [67], (2) prompt engineering use to augment accuracy, and (3) deliberate inclusion of scenarios with variable evidence hierarchies to elicit training differences even in domains where guidelines from recognized organizations are not available. Model evolution necessitates ongoing reassessment.

Although applied to IBD nursing, the framework is generalizable and dynamically adaptable across diverse health care specializations and cross-cultural contexts. Further clinical adversarial testing to detect specific bias, including ethnophysiological bias detection, is detailed in Section A.3 in Multimedia Appendix 2.

The evaluation of consistency over time was conducted using a highly specific, well-defined, and domain-constrained prompt designed to elicit unambiguous answers. In future research, it would be valuable to extend testing the models on responses to a complex clinical case requiring multiparameter considerations also for consistency assessment. Such scenarios are more likely to introduce variability in responses to the same prompt when repeated over time. Moreover, automated consistency assessment relies on semantic similarity embeddings, which do not detect factual drift or safety divergence; thus, expert evaluation of correctness and safety was retained as a separate and primary layer. A specific limitation concerns decoding control. As detailed in the Methods section, this study evaluated models under realistic end user default conditions and not controlled laboratory decoding settings. Future work should replicate the evaluation through direct provider APIs with temperature fixed at 0, a fixed seed where supported, and a prespecified number of 20 samples per prompt, enabling formal separation of decoding variance from genuine response variability.

Future research is planned across 4 primary directions. In health technology, the aim is to expand upon multimodal colearning even from tabular data [68] to learn from diverse data sources (eg, ulcer image, photograph of the error signal from an infusion pump up to signals leading to the biometric identification of the patient, such as an electrocardiogram, heart rate variability index, thermography, and capillaroscopy) to integrate LLMs in developing presymptomatic diagnostic platforms for a wider range of nursing risk assessment scales, while contemplating privacy requirements as mandated by Chapter III of the EU AI Act.

In cybersecurity, the objective is to reinforce alignment mechanisms to prevent jailbreaking, while deterministic XAI techniques can be used to understand the “why” the LLM is behaving like it is and prevent autonomous agent failures.

XAI developments are indicated in a recent systematic review [69], which highlights the “Evolutionary Independent Deterministic Explanation (EVIDENCE) framework” [70], the first deterministic and model-independent method, as “a theoretically grounded and empirically more robust alternative to the heuristic-based approaches of SHAP and LIME” and concludes that “though nascent, it signals a move toward developing more rigorous and specialized XAI techniques” [69]. The goal is to extend the deterministic method to be also problem-agnostic and implement the first counterfactual explanations to obtain algorithmic transparency [71,72].

Transparency obligations applicable to high-risk AI systems are detailed in Article 50 (Chapter IV), and in Article 86 (Chapter IX) of Regulation (EU) 2024/1689. These provisions establish the obligation of transparency for providers and deployers and the data subject’s right “to obtain from the deployer clear and meaningful explanations of the role of the AI system in the decision-making process and the main elements of the decision taken” (XAI), respectively. Furthermore, Recital 133, referring to Article 50, points toward the obligation to make possible the distinction between AI-generated content and authentic human-generated content, through machine-detectable labeling of synthetic content.

The California AI Transparency Act (SB 942) [73] builds its legislative foundations upon it and specifies its exact implementation modality by introducing mandatory watermarking of every AI-generated text or image, technically difficult to remove, incorporated in the file itself (metadata or steganography), and invisible to the human eye (with optional visible marking). The usefulness of watermarking lies in algorithmic deepfakes detection, in the awareness of receiving eventual AI-generated health guidance, and in countering AI-based transformation in research [74], including plagiarism [75].

The use of a locally deployed (on-premise or air-gapped) model—for instance, via the institution’s local hardware (computer) without external connectivity for the inference phase—offers concrete guarantees against the use of uploaded data for training or fine-tuning future model versions (barring guarantees declared by the provider), as well as against the possibility, albeit remote [76], of data breach by hackers or unintentional data leak [77], the latter including instances such as inadvertent transfer of sensitive data by users themselves. Finally, concerns in GenAI span the climate crisis: researchers forecast a 4.2‐6.6 billion m3 depletion of fresh water by 2027 [78], primarily for data center cooling.

Future research should therefore prioritize safe and green architectures by design, allowing local deployment as a convergent solution addressing ecosustainability, operational resilience, democratic access to GenAI in regions without internet coverage, and robust privacy protection for sensitive data, including EHRs processed during the inference phase, across clinical, managerial, and scientific writing in health care.

The proposed methodology will be useful if transformers continue to be mere stochastic memorizers [79], based on the scalar product q⊤k, that is, the statistical probability of co-occurrence. Locally deployable security layers grounded in physical-law representations, combined with the contextual visualization of the internal reasoning paths (algorithmic XAI ready at every interaction) [80], could constitute a high-level guarantee in high-risk settings such as health care.

Findings underscore that domain-specific, regulation-aligned evaluation is essential for high-stakes clinical decision support. The integration of progressive “jailbreaking” techniques within safety assessment revealed a critical dichotomy between a model’s clinical knowledge and its ethical resilience. Anthropic’s Sonnet variants currently represent the most suitable options within our thresholds, meeting key EU AI Act requirements while still requiring expert oversight. Embedding standardized nursing taxonomy improved diagnostic translation and prioritization, suggesting a pathway for integration into EHR-linked workflows. Strengthening health care professionals’ critical thinking during gradual, supervised adoption can enhance rational and intuitive decision-making and standardized terminology competency. The methodological guide supports iterative expert-monitoring cycles for bias identification and continuous model improvement via few-shot exemplars or fine-tuning proposals. Expert oversight, with the human-in-the-loop approach, remains nonnegotiable for both subsequent tuning and usage, even for “recommended” models.

Through the operationalization of EU AI Act requirements for responsible AI in a methodological framework specifically tailored to nursing and health care, dynamically adaptable to clinical specializations, and by extending the investigation to locally functioning SLMs, this study provides a foundational contribution for transversal skills curricula at the international level toward trustworthy, ethical, and sustainable AI integration.