The integration of digital technologies in health care services, especially the implementation of computerized clinical decision support systems (CDSSs), promises to enhance the quality, safety, and efficiency of patient care [1]. Evidence continues to build in favor of implementing CDSS for the optimization of clinical management decisions [2], thereby enabling more effective risk-based decision-making and the delivery of personalized care within the realm of acute health care [3]. Noteworthy applications include the adoption of computerized provider order entry systems [4], the deployment of point-of-care alerts to enhance patient safety [5], and the integration of electronic health record data and artificial intelligence (AI) for decision support [6,7].

Indeed, CDSSs are increasingly incorporating AI and machine learning to realize several critical benefits to clinical decision-making [8]. AI and machine learning techniques facilitate rapid analysis of extensive clinical data, including patient records and medical literature, surpassing traditional rule-based systems in swiftly generating insights and recommendations [1,8]. These technologies excel in identifying intricate patterns and relationships within data, enhancing diagnostic accuracy and treatment recommendations [1]. The relevance of AI and machine learning to CDSS adopters, such as health care providers, lies in the ability to tailor recommendations and predictions to specific scenarios [9]. For instance, machine learning can enhance diagnostic accuracy for common medical conditions by learning from large datasets, which directly benefits the adopters by providing more precise and actionable insights [10]. Although AI-enhanced CDSSs promise more precise, timely, and personalized clinical support, they are designed to complement rather than replace human judgment, necessitating careful consideration of implementation risks and limitations to optimize patient care and outcomes [11].

Despite these considerable potential benefits, the effective implementation of these systems within health services presents multifaceted challenges [1,12]. These include managing diverse stakeholder expectations, emergent clinical and sociopolitical contexts, and changing strategic priorities [13,14]. Failing to address these challenges may give rise to unanticipated outcomes related to low adoption rates [15,16], inappropriate use [17,18], unforeseen consequences [19,20], and long-term technology abandonment [21].

CDSSs can be valuable tools in health care, offering guidance to professionals; however, these systems can face limitations in practice. For example, CDSSs may struggle to accommodate patients with complex comorbidities, potentially leading to treatment recommendations that inadvertently worsen certain conditions such as prescribing heart disease medication that could harm kidneys [1]. Furthermore, CDSSs often do not incorporate patient preferences, cultural beliefs, or financial constraints as these data sources are not considered, necessitating personalized adjustments by clinicians to ensure treatment adherence and efficacy [1]. In acute care settings, CDSS may not keep pace with rapidly evolving clinical conditions, requiring clinicians to rely on their real-time assessments rather than potentially outdated CDSS guidance [1,9]. Moreover, there are high costs associated with the implementation, adoption, and maintenance of CDSSs [22,23]. These limitations of CDSSs underscore the critical role of effective implementation so that the right CDSS can provide the right information at the right time to the right patient [1].

Examining the factors that drive the successful adoption of CDSSs and those that impede its progress contributes to a deeper understanding of the dynamic interplay between technology and health care delivery [24-26]. This can be approached systematically [1,12-21] with guidance from well-established theories within the discipline of implementation science, which also accounts for organizational complexity [27,28]. A recent scoping review found that models, rather than theories or frameworks (18/42, 43% of the included studies), were most frequently used to guide CDSS adoption and evaluation strategies [29]. Unlike frameworks, models can be limited in examining the complexity of sustained implementation, acceptability, and adoption of technology across organizational and system levels [29,30]. The Non-Adoption, Abandonment, Scale-Up, Spread, and Sustainability (NASSS) framework uses a complex systems approach, which encapsulates the determinants NASSS of technological adoption overtime in health care settings [24]. This framework serves as a conceptual lens through which technological interventions are viewed as part of a complex system consisting of many processes and components. The utility of the NASSS framework lies in its ability to identify contextually appropriate determinants and inform implementation strategies, thereby shedding light on the factors that impact the success of digital health implementations [31-33]. Contextually informed implementation strategies tailored to influence clinician behavior could have a greater influence on CDSS adoption than technological design and content features [15,24-26,34].

As such, this research was characterized by 2 principal aims. First, we aimed to identify the NASSS theory-informed determinants that either facilitate or hinder the application of CDSSs within a metropolitan health service. These include multiple CDSS interventions at the health-service level and at a single hospital setting spanning for 2 years. Second, this study aimed to map these determinants into specific stages of the implementation process, thereby developing a systems-level understanding of CDSS application across implementation stages. A stage-specific mapping approach allows for more nuanced and tailored strategies for CDSS integration, ensuring that the unique challenges and opportunities associated with each implementation stage are addressed [35-37].

The Metro South Health Human Research Ethics Committee granted ethical clearance for this research (HREC/2020/QMS/64807). All participants provided written and verbal informed consent before participating in the study. Participation was voluntary and participants could withdraw at any time. All data were deidentified and handled in accordance with the Metro South Human Research Ethics Committee guidelines.

This qualitative study used a hybrid inductive-deductive approach [38]. The deductive approach was informed by the NASSS framework to identify contextually specific determinants associated with the use of CDSS technology in a large metropolitan health service across discrete implementation cycles during a 2-year period. The NASSS framework positions technological interventions as part of a complex system and has been used to guide implementation efforts and identify factors that influence technology implementation success in health services [24,31-33].

This study was conducted in a metropolitan health service comprising 5 hospitals, which serve a large catchment area (3856 km²) in Australia. Consequently, those employed by the health service within the digital health and informatics portfolio were recruited to take part in the study. It must be noted that this department operates at both a health-service level and a facility level. In addition, clinicians who worked at either of the 2 largest hospitals within the health service were also recruited. With a facility of 1033 beds, hospital 1 is the largest teaching and training university hospital and is equipped with all major medical specialties except maternity services and pediatrics. Hospital 2 comprises most medical specialties, with 459 beds, including maternity services and pediatrics. Both hospitals (and the health service at large) used the same integrated electronic medical record system that had been implemented before the commencement of the study.

The participants sampled in this study included staff from the clinical informatics unit, which operated at the health-service level. In addition, clinical staff who worked within the 2 largest (defined according to the number of available beds) hospitals within this health service were also recruited to participate in the study. In this study, CDSSs were defined as any electronic system or interface designed to provide health system users with tailored information to inform decision-making within a particular context or situation [1,31]. The participants who met the following selection criteria were recruited for this study: experience with decision-making, governance, purchasing, design, and implementation of CDSS initiatives within the health service. This could include those with roles in informatics, governance, and management, as well as frontline clinical staff. The researchers used purposive sampling throughout the study, seeking representativeness of participants who were involved in using or implementing CDSS. The researchers also sought representativeness across a range of implementer roles, that is involved in making decisions or engaging in CDSS procurement, rollout, and upgrades, and adopter roles, that is clinician users of the CDSS [39,40]. To obtain this representativeness, the researchers estimated a sample size of 30 to 40 participants.

Acting as a knowledge broker, SM, an academician who is also embedded in the health service, used a knowledge brokering process [41] to identify potential participants during informal discussions with health-service staff. Those identified were then formally invited to the study using internal memos and emails sent from SM, with participants given a week to respond. Nonresponders were followed up one more time within a fortnight of the initial email. Saturation was deemed to be achieved when no new concepts or understanding were identified after 3 consecutive interviews following purposive sampling [39].

The reporting of findings was guided by the Standards for Reporting Qualitative Research checklist (Multimedia Appendix 1). A semistructured, NASSS framework–informed interview guide, which focused on the availability, development, and perceptions of CDSS within the participants’ health system (Multimedia Appendix 2), was developed. Questions also explored decisions around the implementation of CDSS as well as their adoption, use, and sustainment in the course of time.

Data collection was conducted between March 2021 and March 2023. On the basis of the participants’ preferences and availability, data were collected through one-on-one interviews or through focus group discussions among groups consisting of 6 participants. This was done through in-person and web-based (Teams; Microsoft Corporation) techniques. Interviews and focus group discussions were led by SM, an experienced digital health and health services researcher (male, PhD qualified, embedded in health service, and familiar to a few participants), and SN, an experienced mixed methods health services researcher (male and PhD qualified). The interview duration was approximately 1 hour, while focus group discussions were conducted for up to 2 hours. The interviews and focus group discussions were audio recorded and transcribed verbatim. Preliminary findings were monitored and discussed with the research team. This information was used to guide recruitment until saturation was reached [33,42].

Initially, SN analyzed the transcripts thematically and concurrently with data collection to facilitate purposive sampling and identify saturation. Early analysis of transcripts allowed us to identify preliminary themes and gaps, informing subsequent interviews and ensuring that a diverse range of perspectives were explored [33,42,43]. This iterative process of concurrent analysis and interviewing was integral to refining our understanding and ensuring the trustworthiness of the findings [42,44]. At the completion of the interviews and focus group discussions, MF and SN analyzed data using qualitative content analysis [31] and framework analysis [31,45-47]. Framework analysis is a systematic qualitative analysis widely used in health and social care research to organize and analyze large volumes of textual data, enabling comparison by case and theme [48].

The data analysis process is outlined in Figure 1. First, barriers, facilitators, and neutral factors associated with CDSS across implementation cycles were assigned a short summarizing phrase [31,42]. This was done through inductive open coding of each transcript. These initial phrases were discussed and revised, with a few being discarded, amended, or subsumed to create higher-order codes [31]. Through an initial inductive analysis of codes, the researchers aimed to capture the context-specific elements of the data without prematurely applying a predefined framework [38,46]. This approach enabled the discovery of preliminary themes and patterns, which were subsequently aligned with the NASSS domains and subdomains during the deductive phase of the analysis [33,38,46].

As a part of this process, where appropriate, illustrative quotes were extracted for each of these codes [42]. The selection of illustrative quotes during qualitative analysis was guided using a consensus approach after discussion with MF, SN, BA, and ZT. The selection of quotes aimed to capture diverse perspectives and enrich subsequent mapping to the NASSS framework to showcase the variations within each theme and subtheme, ensuring that findings were both grounded in the data collected and demonstrative of the range of perspectives expressed by the participants [33,45,46,48-50]. The outcome of this process was to reflexively map each of the higher-order themes onto ≥1 of the NASSS domains and subdomains with which they were aligned [31]. This analysis was primarily deductive, with the intent to align barriers, facilitators, and neutral factors identified in our inductive analysis with preexisting domains within the NASSS framework [31]. It is important to note that, in the analysis, no restriction was placed on the number of domains or subdomains with which an individual-coded barrier, facilitator, or neutral factor could align [31]. This was not only to categorize using the NASSS framework but to reflect on the role of the theoretical framework in capturing the complexity of CDSS implementation within a real-world health service [31,51]. Themes not aligned to an NASSS domain or subdomain were coded separately [46]. Finally, the Active Implementation Framework [35,36] was used to categorize and display identified determinants according to the stages for digital health implementation. These stages include exploration, which is focused on the feasibility and organizational readiness, installation, which is centered on the organizational preparation, initial implementation, which covers implementation-initiation techniques such as training, and full implementation, which emphasizes sustainment [35-37].

A descriptive numerical summary of the identified determinants was collated in an Excel (Microsoft Corporation) spreadsheet, mapping determinants to the NASSS framework domains and the Active Implementation Framework stages [31,42]. To illustrate the alignment of barriers, facilitators, and neutral factors with the NASSS framework and the Active Implementation Framework, the researchers developed visual matrices to depict the frequency and salience of the identified themes and their association with various implementation phases [31]. This visual and numerical representation aimed to demonstrate common findings across the NASSS domains relating to CDSS implementation and to identify gaps in the implementation of CDSS across different phases.

Throughout the analysis, the researchers maintained trustworthiness and reflexivity by reflecting on the research process through discussion as a research team and considering how their perspectives and the chosen frameworks may have influenced data interpretation [45-47,49]. This study provided a narrative report of the framework mapping to each NASSS domain and subdomain supported by direct quotes from the interviews and focus groups [33,45,46] and discussed the implications for practice.

A total of 30 participants, including implementers, decision makers, and CDSS end users, across several departments took part in this study. Table 1 provides comprehensive demographic information about the participants.

Participants were mapped to NASSS-informed implementation roles, highlighting their level of decision-making within the hospital systems of interest, as shown in Figure 2. Of the 30 participants, 14 (47%) clinical staff identified solely as adopters of CDSS tools. In addition, 17% (5/30) of clinical staff took on the role of organizational staff champions, that is, clinicians who advocate for the technology and its use [52,53]. Overall, 17% (5/30) of participants were informatics professionals, 13% (4/30) of participants acted in dual roles of organizational staff leadership and clinical adopters, that is, as clinical directors. Only 1 (3%) participant encompassed adopter-clinician and organizational staff–implementer informatics roles, that is, as a clinician who worked within the digital health and informatics directorate portfolio. Only 1 (3%) participant was considered a wider system–interorganizational networker.

The identified determinants encompass specific elements pertinent to CDSS implementation. These elements were classified into barriers, enablers, and neutral factors. For a comprehensive breakdown of these determinants, refer to Multimedia Appendix 3. The findings were mapped to the NASSS domains and subdomains, as shown in Figure 3. Barriers (71/126, 56.3%), enablers (50/126, 39.6%), and neutral factors (5/126, 3.9%) associated with CDSS implementation were identified in this study. Most determinants were mapped to the organization (39/126, 30.9%), technology (32/126, 25.3%), and adopter (25/126, 19.8%) domains.

By mapping NASSS-informed determinants influencing CDSS implementation cycles using the Active Implementation Framework stages [35-37], our research provides pragmatic insights to inform tailored integration strategies of these technologies into large hospital systems. Despite encountering limitations in implementation capacity and planning know-how, the institution demonstrated strength through a well-equipped clinical informatics workforce and early-stage training of its adopter workforce. This finding aligns with existing literature, emphasizing the critical role of organizational culture and support structures in successful technology adoption [13,31]. Second, CDSS implementation faced technological hurdles, including interoperability and interface issues, a frequently reported CDSS engagement barrier [15,22,55]. However, positive user experiences emerged as a significant factor influencing CDSS use, particularly when adopters could directly experience improved efficiency and better patient outcomes for themselves. This underscores the importance of user-centric design and showcases the practical impact of CDSS on health care workflows, aligning with the broader pattern of emphasizing end user perspectives in technology adoption [12,15,17]. The presence of peer champions among clinicians emerged as a significant enabler. Peer-to-peer support and advocacy played a crucial role in enhancing clinician confidence and acceptance of CDSSs, contributing to a smoother initial implementation process. Moreover, the study reveals wider systemic barriers, such as a lack of clear governance for CDSS applications, aligning with the ongoing discourse on the necessity for robust regulatory frameworks in the broader implementation of digital health care technologies [25,56]. This finding emphasizes the need for a systemic approach to address governance gaps and ensure the effective integration of CDSSs into health care systems. These findings strongly align with our recent scoping review, which mapped reported CDSS implementation barriers and enablers to NASSS domains [31]. Although further empirical evidence from successful implementation is required, this further highlights the reliability of the NASSS framework as a pragmatic tool to identify meaningful domain-level determinants associated with the implementation of technology within health systems [24,54].

When determinants were mapped to the Active Implementation Framework stages [35-37], a clear pattern emerged highlighting implementation activities most notable in the early stage of exploration and the later stage of full implementation. The absence of planned piloting and process evaluation was not unexpected, given a lack of evaluation and implementation planning know-how was identified as a key organizational readiness barrier.

This may be addressed through the application of theory-informed approaches to implementation planning [27,30]. This finding aligns with the gaps identified in our recent scoping review examining the use of theories, models, and frameworks to support CDSS implementation within hospital systems [29]. This review [29] reported an inconsistent application of systematic approaches to implementation planning within hospital systems in several countries.

Furthermore, while organizational-level training of champions was crucial and supported CDSS adoption in both the exploration and installation stages, this was not sustained throughout the implementation life cycle. This weakening of the informatics-adopter sociotechnical network during the course of time may have contributed to the communication issues adopters frequently highlighted. This study identified 3 key communication issues in the implementation of CDSSs. First is the challenges in effectively communicating and coordinating within the complicated web of governance structures, potentially impeding the progression of CDSS implementation. Second, the limitations in ongoing communication about system updates and advancements may lead to adopters being unaware of improvements, impacting their ability to fully use the CDSS. Third, trust in CDSSs was recognized as varying among clinicians, indicating a potential communication challenge in conveying the benefits and reliability of the system uniformly across all CDSS adopters. More rigorous implementation planning, encompassing all 4 stages, may go some way in addressing the identified communication gaps [27,29,30,35,36].

Furthermore, this study identified the need for ongoing adopter support and planned evaluation strategies encompassing the full implementation cycle [36,56]. Evidence indicates that sustained cycles of monitoring and iteration may lead to sustained integration of CDSS in health care systems [21,34,57]. For example, planned mechanisms for user tracking allow for real-time assessment of uptake, which may impact the choice and delivery of implementation strategies to facilitate the continuous adoption of the innovation in practice [10,58].

The study’s strengths lie in its holistic systems thinking approach, which provides a comprehensive understanding of the challenges and enablers within each implementation stage and across implementation cycles for complex systems such as CDSS. The researchers acknowledge that the final phase, defined as full implementation within the Active Implementation Framework, does not fully address all components of sustainability, which can include factors such as changing legislation, government (federal and state) budgets, and workforce capacity [35,36,57,59]. However, this framework does acknowledge factors that bridge the policy gap within an organization, including clear governance pathways and longer-term resourcing, in addition to recognizing embedded behavior change and ongoing monitoring and iteration as parts of full implementation cycles [35].

It must be noted that while this study was intentionally designed for a 2-year period, there may have been specific events (including the COVID-19 pandemic) that may have influenced participant perceptions. However, it was not within the study’s scope to contextualize a comprehensive list of events that could have impacted perceptions. Nonetheless, our questions were designed to probe reasons behind participant perceptions when interviewed; early analysis of transcripts allowed us to identify preliminary themes and gaps, ensuring that a diverse range of perspectives were explored and stopped when no new insights were gained [33,42,44]. The participants’ roles can influence their perceptions and experiences with CDSS implementation. Clinical adopters might focus more on usability and patient impact, whereas senior management or informatics professionals may prioritize technical challenges and integration issues [60]. Findings may not fully generalize to other health care settings with different organizational structures, levels of technological maturity, or cultural contexts [10]. It must be noted that patient perspectives were outside the study scope, which focused on clinicians as end users within an acute-care (hospital) setting. Recommendations and insights derived from this study may need to be interpreted cautiously in broader contexts and may apply more directly to the study settings where similar participant distributions and roles are prevalent [10,61]. Future work should also consider examining patient perspectives to enhance insights.

The study was conducted within a specific health care system, potentially limiting the generalizability of some findings. However, the use of the NASSS framework has identified multilevel determinants that align with trends in contemporary research findings across the health care system [31,62,63]. Moreover, the study’s participant population represents a broad spectrum of stakeholders involved in CDSS implementation, including clinical staff adopters, organizational champions, and informatics professionals. This diversity ensured the representativeness of perspectives needed to understand the broader contextual factors applicable beyond this study’s settings [63,64]. By including individuals from various departments and decision-making levels, the study captures a holistic view of the challenges and facilitators influencing CDSS adoption, contributing to a nuanced understanding of implementation dynamics within the hospital system. Furthermore, this qualitative study was exploratory in nature and enabled us to unpack the intersectionality of multiple determinants in influencing a range of implementation outcomes.

Future research could extend the use of the NASSS framework to the application of CDSS in different health care environments or medical conditions such as in rural hospitals or mental health care [31,34]. Furthermore, investigations might integrate a systems-level framework, such as the NASSS framework, using behavioral assessments, standardized psychological tests, or multimodal methodologies to explore individual emotional reactivity for CDSS adoption [13,65]. Such endeavors may deepen insights into health care professionals’ emotional responses, such as stress levels, satisfaction, and confidence [13], to CDSS use. The insights gained could contribute to broader knowledge about technology implementation across diverse health care contexts, informing tailored strategies for improved adoption, long-term viability, and best practices to enhance CDSS adoption in various health care settings [31,66]. Additional quantitative data and further empirical application of the NASSS framework may be beneficial in further exploring specific qualitative findings identified in this study. Future research could explore the longitudinal aspects of CDSS adoption to capture changes and adaptations over extended periods by conducting trials with integral process evaluations to test the identified implementation factors across diverse populations and settings [53,60,63].

These findings address a significant knowledge gap in the literature using system thinking principles to identify the interdependent dynamics of CDSS implementation. In moving forward, this study serves as a catalyst for informed decision-making in CDSS implementation, offering actionable insights for practitioners and researchers alike. As technology continues to evolve and health care landscapes transform, the lessons gleaned from this study provide a foundation for refining CDSS implementation strategies and advancing patient-centered, efficient, and ethically sound health care practices.