This scoping review was conducted according to the Joanna Briggs Institute methodology and reported following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist. Ethics approval was not required as all data used were published prior to study initiation.

A detailed protocol for this review has been published previously [3]. In brief, potential evidence sources were identified by searching 5 electronic databases (MEDLINE, Embase, CINAHL, APA PsycInfo, and the Cochrane Library) for articles describing clinical decision support, implementation and/or determinants, and AI or ML published from database inception until May 10, 2022. These databases were selected to capture clinical and health service literature relevant to the real-world implementation of AI-based CDSSs. Given the focus on implementation within clinical practice environments, databases indexing biomedical and health service research were prioritized. Technical and engineering databases were not included as they predominantly index studies focused on algorithm development, model performance, or technical validation rather than real-world clinical implementation.

Key search concepts and corresponding keywords are listed in Table 1. The full electronic search strategy for MEDLINE (Ovid) is provided in Table S1 in Multimedia Appendix 1 [10-18]. Search strategies for the other databases were adapted from this core strategy using database-specific operations and syntax. The titles and abstracts of the articles returned from the search were screened for potential relevance by 3 independent reviewers, requiring agreement from at least 2 reviewers to be included for full-text screening. The full-text screening and selection of final articles was carried out by 2 independent reviewers according to the eligibility criteria described in the following section. Disagreements were handled by a third reviewer who participated in the initial abstract screening. Reasons for exclusion at the full-text stage were recorded and are summarized in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram.

Primary research describing the implementation process or reporting determinants (barriers and facilitators) of implemented AI-based CDSSs was considered eligible for inclusion. AI was defined as any computerized system with capabilities for perceiving, synthesizing, and inferring information in a humanlike manner, encompassing both traditional rule-based and modern data-driven techniques [1]. Clinical decision support was defined as the generation of actionable diagnostic or prognostic insights intended to aid clinicians in making personalized decisions about patient care, which resulted in the exclusion of studies focusing on nondecision task automation, for example, image segmentation and recognition tasks in radiology. Studies were also excluded if they described non-AI-based CDSSs or solely patient-facing aids or if they reported on the development and/or effectiveness of AI-based CDSSs but did not describe their implementation. Eligibility was not limited by study design or language. A summary of the inclusion and exclusion criteria is provided in Table 2.

Consistent with scoping review methodology and the objective of mapping the breadth of available evidence, a formal assessment of risk of bias or methodological quality of the included studies was not conducted.

Bibliometric information, study characteristics, reported determinants, and implementation details were abstracted from the articles selected for inclusion using a standardized data extraction form developed and piloted by 2 independent reviewers. Prior to full data extraction, reviewers completed a calibration exercise using a subset of included studies to ensure consistency in data abstraction. A split-half approach was used whereby each reviewer extracted data from a subset of studies and cross-checked the extracted data to identify and resolve discrepancies.

All abstracted determinants were mapped to the updated Consolidated Framework for Implementation Research (CFIR) to inform our analysis of implementation experiences [19]. The CFIR was selected as it provides a comprehensive and widely used framework for systematically categorizing multilevel determinants influencing implementation in health care settings [19,20]. The CFIR contains 5 main domains: innovation, inner setting, outer setting, individuals, and implementation process. The innovation domain refers to characteristics of the system being implemented. The inner setting and outer setting domains refer to the setting in which the innovation is being implemented (eg, clinical unit or department) and the setting in which this inner setting exists (eg, hospital or health system), respectively. The individuals domain refers to characteristics of the individuals involved in implementation (eg, leaders, innovation deliverers, and innovation recipients), and the implementation process domain describes the activities and strategies being used to implement the innovation.

Determinants were identified by reviewing the full texts of the included studies to extract explicit statements describing factors that influenced the implementation of AI-based CDSSs. Extracted data consisted of verbatim quotations or clearly stated author interpretations reflecting barriers to or facilitators of implementation.

Two reviewers independently identified and extracted eligible determinant statements and classified them as barriers or facilitators based on whether they were described as hindering (barriers) or supporting (facilitators) implementation processes or outcomes. These determinants were then mapped to CFIR domains and constructs. Mapping decisions were based on the underlying meaning and implementation context of each extracted statement, with determinants assigned to the CFIR construct that most closely reflected the primary implementation-related factor described. General descriptions of system features, usability outcomes, or design characteristics were not considered determinants unless they were explicitly linked to implementation processes or outcomes. Discrepancies were resolved through discussion and consensus. Only studies reporting explicit implementation determinants were included in the CFIR-based synthesis.

A total of 15,109 evidence sources were identified in our initial search, of which, after removal of 4234 (28.0%) duplicate records, 10,875 (72.0%) studies were screened based on title and abstract. Following title and abstract screening, of the 10,875 screened studies, 10,355 (95.2%) were excluded. After removing based on full text availability, 494 (4.5%) articles were assessed for eligibility. Of these 494 articles, 13 (2.6%) met the inclusion criteria and were included in the final review (Figure 1). Study-level characteristics for each included source of evidence are summarized in Table 3.

Most excluded studies did not report on system implementation at all or did not describe it in sufficient detail (Figure 1). Nearly all included articles (11/13, 85%) were published between 2019 and 2022 by scholars from the United States. Over half (7/13, 54%) of the selected studies discussed AI-based CDSSs developed for use in critical care (ie, sepsis or delirium) and emergency medicine settings, and most systems (8/13, 62%) were ML based and implemented at a multicenter scale (Table 3).

Of the 13 included studies, 9 (69.2%) explicitly reported implementation determinants (barriers and/or facilitators) and were therefore included in the CFIR mapping. The remaining 4 (30.8%) studies did not meet the criteria for explicit determinant extraction as they focused primarily on system design, usability, or general implementation considerations without clearly identifying barriers or facilitators tied to the implementation process. Extracted quotations, CFIR mappings, and barrier and facilitator classifications are provided in Table S2 in Multimedia Appendix 1 [10-18].

Despite a recent increase in publications describing the development of AI-based CDSSs, successful real-world implementations in health care remain limited. Given their potential to improve the quality of patient care and reduce clinician burden, a better understanding of how to effectively implement these tools is needed [8].

This scoping review identified several key determinants from studies describing real-world implementations, including the robustness of the evidence base supporting efficacy claims, the availability of sufficient IT infrastructure, how well the system can be integrated with current clinical workflows and processes, and the overall complexity of the system. Assessing clinician needs, outlining and reviewing implementation plans, forming multidisciplinary implementation teams, encouraging clinician adoption and participation in implementation, and providing performance-based feedback were also found to be primary facilitators of implementation. Conversely, barriers may be faced when clinician end users and/or implementation leads lack the capabilities or motivation to fully commit to the project. Other determinants included the generalizability of the AI models to different patient populations, the relative advantage offered by the system compared to current practice, access to training sessions and educational materials for users, data availability, and shared values about clinician and patient needs.

Notably, only a small number of studies reported explicit implementation determinants, highlighting a critical gap in the literature. This suggests that many real-world implementations of AI-based CDSSs may not adequately evaluate or report factors influencing adoption and sustained use. Addressing this gap will be essential for advancing the translation of these systems into routine clinical practice.

The determinants identified in this review can be used to guide future implementation projects through the designation of appropriate strategies that mitigate barriers and leverage facilitators to successfully implement AI-based CDSSs. Our results support the use of several strategies derived from the Expert Recommendations for Implementing Change project, which aimed to synthesize knowledge from a wide range of implementation science and clinical practice experts [37].

First, developing and distributing educational materials explaining how the specific AI-based CDSS works, detailing the results of clinical trials or other external validation tests, and describing the patient population the system was developed with may help reduce barriers related to the innovation’s complexity and evidence base. Adopting an integrated knowledge translation approach that directly involves end users in both the design and implementation of new innovations could help mitigate these barriers further upstream [38]. Conducting educational meetings targeted toward different stakeholder groups, including clinicians, patients, caregivers, hospital administrators, and IT staff, to teach them about the innovation and its clinical benefit may also help reduce initial resistance to uptake [37].

End user resistance can also be eased through the identification of clinical champions; these individuals encourage uptake by raising awareness and providing motivation and leadership to their peers [37,39]. Uptake can be further encouraged through the influence of local opinion leaders, who should be identified and informed about the benefits of the innovation prior to its implementation [37].

Finally, implementation teams should aim to capture and share knowledge throughout the implementation process. Regular engagement sessions with clinician end users allowing for reflection on implementation efforts can be used to collect information about what specific strategies worked in their local context, which can later be disseminated to help inform future implementations of AI-based CDSSs [37].

The determinants identified in this scoping review, particularly those related to algorithm interpretability, data availability and quality, and workflow integration, are already well documented in the literature [40,41]. However, many of these studies identified barriers and facilitators based on anticipated implementation challenges prior to real-world deployment. Thus, our focus on accounts of real-world implementation represents a major strength of this scoping review, bridging the gap between theory and practice to provide actionable insights.

Furthermore, the determinants identified in this review are not limited to a single clinical domain, as is often the case with previously published work in this area [41,42]. Synthesizing determinants across a broader clinical context allows for more generalizable considerations that can be applied to a wider range of future CDSS implementations.

Recent interview-based work exploring stakeholder perspectives on improving AI-based CDSSs and their integration into care identified similar challenges related to workflow integration, evidence quality and transparency, clinician trust, training and education, and organizational readiness [43].

Although our findings are well supported, there are some determinants commonly cited in the literature that were not reported in the articles included in this scoping review. Notably, there was no mention of the effects of cost and willingness to pay, which are known to be key drivers of success for health care innovations [44]. This could potentially be a result of our focus on real-world implementations as these projects would likely have undergone a cost-benefit analysis prior to initiation at a given hospital or health system. However, it is important to note that AI-based CDSSs in particular require additional investments to deploy and maintain extensive IT infrastructure, as well as providing training for end users; these costs must be balanced by an improvement in the quality of patient care for implementation to be supported by hospitals and health systems [27,45,46].

Finally, consistent with our finding that only a limited number of studies explicitly reported implementation determinants, a recent scoping review examining patient-related benefits of AI-based CDSSs in sepsis care also emphasized the need for further implementation-oriented and prospective research to support real-world clinical integration [47].

This study has several limitations. First, this review adopted a broad definition of AI, encompassing both rule-based and data-driven systems. While this approach enabled the inclusion of a larger body of literature and a broader range of implementation determinants, the findings may not be specific to contemporary AI-based implementations. Therefore, some identified determinants may reflect broader CDSSs or health IT implementation considerations.

Second, our strict definition of clinical decision support and focus on real-world implementation studies resulted in a narrow scope and a limited number of included studies, which constrained the depth of analysis. As a result, unsuccessful, aborted, or prerollout implementation attempts may have been underrepresented despite their potential to provide important insights into barriers preventing real-world adoption. However, we feel that this approach was justified by our aim to address the knowledge gap pertaining to the real-world implementation of AI-based CDSSs. In addition, among the included studies, most did not report implementation outcomes in sufficient detail to identify determinants that were most influential to the successful adoption, integration, and sustained use of AI-based CDSSs over time.

Third, most included studies were conducted in the United States and within selected clinical contexts, which may limit the transferability of findings to other health care systems, organizational structures, and clinical settings.

Fourth, although our search strategy focused on major biomedical and health sciences databases, it is possible that relevant studies published in technical or engineering venues were not captured. Given our focus on real-world clinical implementation, we anticipate that most eligible studies would be indexed in clinically oriented databases.

Fifth, there was a limited representation of patient perspectives in the included studies. While this may partially reflect our focus on clinician-facing tools, understanding how patients perceive the use of AI-based CDSSs in their care remains an important area for future research.

Finally, this review is subject to temporal limitations. The literature search was conducted in 2022, and given the rapid evolution of AI in health care, additional relevant studies may have been published since that time. As such, the findings should be interpreted as reflecting the evidence base available at the time of the search.

This review identified key determinants influencing the real-world implementation of AI-based CDSSs, emphasizing the importance of system design, organizational context, and implementation strategies. However, the limited number of studies reporting explicit implementation determinants underscores a critical gap in the literature. Future work should prioritize the systematic identification and reporting of implementation determinants to better inform adoption and sustainability. These findings provide a foundation for developing targeted implementation strategies and highlight the need for more rigorous, implementation-focused research to support the successful integration of AI-based CDSSs into routine clinical practice.