This paper is in the following e-collection/theme issue:

Reviews in Medical Informatics (82) Decision Support for Health Professionals (1127) Clinical Information and Decision Making (1314) Clinical Informatics (904) Ontologies, Classifications, and Coding (223)

Published on in Vol 11 (2023)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/43053, first published .
Ontologies Applied in Clinical Decision Support System Rules: Systematic Review

Ontologies Applied in Clinical Decision Support System Rules: Systematic Review

Ontologies Applied in Clinical Decision Support System Rules: Systematic Review

Authors of this article:

Xia Jing1 Author Orcid Image ;   Hua Min2 Author Orcid Image ;   Yang Gong3 Author Orcid Image ;   Paul Biondich4 Author Orcid Image ;   David Robinson5 Author Orcid Image ;   Timothy Law6 Author Orcid Image ;   Christian Nohr7 Author Orcid Image ;   Arild Faxvaag8 Author Orcid Image ;   Lior Rennert1 Author Orcid Image ;   Nina Hubig9 Author Orcid Image ;   Ronald Gimbel1 Author Orcid Image
Article Authors Cited by (3) Tweetations (2) Metrics

    Review

    1Department of Public Health Sciences, Clemson University, Clemson, SC, United States

    2College of Public Health, George Mason University, Fairfax, VA, United States

    3School of Biomedical Informatics, The University of Texas Health Sciences Center at Houston, Houston, TX, United States

    4Clem McDonald Biomedical Informatics Center, Regenstrief Institute, Indianapolis, IN, United States

    5Loweswater Consulting, Combria, United Kingdom

    6Ohio Musculoskeletal and Neurologic Institute, Ohio University, Athens, OH, United States

    7Department of Planning, Aalborg University, Aalborg, Denmark

    8Department of Neuromedicine and Movement Science, Norwegian University of Science and Technology, Trondheim, Norway

    9School of Computing, Clemson University, Clemson, SC, United States

    *these authors contributed equally

    Corresponding Author:

    Xia Jing, MD, PhD

    Department of Public Health Sciences

    Clemson University

    519 Edwards Hall

    Clemson, SC, 29634

    United States

    Phone: 1 8646563347

    Email: xjing@clemson.edu


    Background: Clinical decision support systems (CDSSs) are important for the quality and safety of health care delivery. Although CDSS rules guide CDSS behavior, they are not routinely shared and reused.

    Objective: Ontologies have the potential to promote the reuse of CDSS rules. Therefore, we systematically screened the literature to elaborate on the current status of ontologies applied in CDSS rules, such as rule management, which uses captured CDSS rule usage data and user feedback data to tailor CDSS services to be more accurate, and maintenance, which updates CDSS rules. Through this systematic literature review, we aim to identify the frontiers of ontologies used in CDSS rules.

    Methods: The literature search was focused on the intersection of ontologies; clinical decision support; and rules in PubMed, the Association for Computing Machinery (ACM) Digital Library, and the Nursing & Allied Health Database. Grounded theory and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines were followed. One author initiated the screening and literature review, while 2 authors validated the processes and results independently. The inclusion and exclusion criteria were developed and refined iteratively.

    Results: CDSSs were primarily used to manage chronic conditions, alerts for medication prescriptions, reminders for immunizations and preventive services, diagnoses, and treatment recommendations among 81 included publications. The CDSS rules were presented in Semantic Web Rule Language, Jess, or Jena formats. Despite the fact that ontologies have been used to provide medical knowledge, CDSS rules, and terminologies, they have not been used in CDSS rule management or to facilitate the reuse of CDSS rules.

    Conclusions: Ontologies have been used to organize and represent medical knowledge, controlled vocabularies, and the content of CDSS rules. So far, there has been little reuse of CDSS rules. More work is needed to improve the reusability and interoperability of CDSS rules. This review identified and described the ontologies that, despite their limitations, enable Semantic Web technologies and their applications in CDSS rules.

    JMIR Med Inform 2023;11:e43053

    doi:10.2196/43053

    Keywords

    clinical decision support system rules; clinical decision support systems; interoperability; ontology; Semantic Web technology 


    For more than half a century, clinical decision support systems (CDSSs) have been developed and used in clinical care delivery [1-5]. Some early CDSS examples include Dialog [6], INTERNIST-1 [7-9], Quick Medical Reference [8], and Iliad [10-12]. The effectiveness of CDSSs in clinical care has been established [13-15], with some pioneering researchers’ work on CDSS effectiveness particularly noteworthy [16]. Researchers have examined CDSS users’ and developers’ experiences, discussed their CDSS vision for the future [17], and recommended best practice guidelines in CDSSs [18-22]. Meanwhile, the challenges of CDSSs have been well documented [23]. Meeting clinician information needs is one way a CDSS can help health care providers improve clinical care quality. Many studies, such as Infobutton [13,24], have demonstrated the effectiveness of CDSSs in this aspect. CDSSs are currently routinely used in clinical care, with rates ranging from 68.5% to 100% in primary care settings based in offices [25] in the United States as part of electronic health record (EHR) systems. CDSSs can take many forms, including but not limited to reminders for preventive services (eg, immunizations and screening tests) [26-28], alerts for drug-drug interactions [22,29,30], diagnostic or treatment plan recommendations [31-33], clinician content assistance [34-38], and recommendations for adhering to current clinical practice guidelines [39-41]. CDSSs have played an important role and are widely used in practice to provide safer and better clinical care services.

    CDSS rules, which function similarly to the human central nervous system, direct the behaviors of a CDSS during operations by incorporating patient data, contextual information, and medical domain knowledge. The central role of CDSS rules is a decisive factor in the relevance and usefulness of a CDSS in the overall clinical workflow, which impacts whether a CDSS is adopted and routinely used. CDSS rules can be written in Arden syntax [42], Semantic Web Rule Language (SWRL), Jess, Jena, and other programming languages, and the processes are labor intensive. Only specially trained personnel are qualified to write such rules. Moreover, regular updating of CDSS rules is required to keep CDSSs relevant and useful in clinical care delivery. However, the process of developing, updating, and maintaining CDSS rules is time-consuming and resource intensive [4,43], making it difficult for both large institutions and resource-constrained small-scale practices. CDSS rule usage data, such as rule fire rates, overwrite rates, successful rates, and user feedback data, can be collected to improve and customize CDSSs and manage CDSS rules. Typically, CDSS rule maintenance entails adding, deleting, and updating CDSS rules.

    Ontologies have been successfully applied to generate and supply domain knowledge in the use, reuse, sharing, and interoperability of information. Ontologies are seen as promising solutions to the challenges of managing and maintaining CDSS rules across institutional boundaries. The Semantic Web is a technology enabled by ontology [44] that is critical in information sharing and reuse [45,46], medicine [47], and CDSSs [48,49]. Although there are numerous definitions of ontology, we used Gruber’s definition in this manuscript: “an ontology is a specification of conceptualization” [45]. Interoperability has been identified as a major challenge for health care information technologies, particularly when it comes to sharing health information across institutional or national boundaries. Ontologies have the potential to shorten the interoperability gap.

    Reusing and sharing CDSS rules are important, but they are not yet routine operations; thus, we conducted this systematic literature review. This study aims to expand on the current state of using ontologies in CDSS rules by conducting a systematic review of the literature on the intersection of CDSS rules, Semantic Web technologies (particularly ontologies), and use of ontologies in CDSSs. The review is expected to provide a comprehensive view of using ontologies in CDSS rules, with granular details. The results could serve as a basis to form a knowledge framework of the topic that may inspire future research. The research question we intend to answer with this systematic literature review is as follows: What is the current state of using semantic technologies, particularly ontologies, to leverage CDSS rule interoperability? Furthermore, the manually annotated results of selected publications could serve as gold standards for automatically identifying relevant entities in the literature.


    Databases and Search Strategies

    Figure 1 illustrates the general workflow we used to conduct this literature review. An initial set of literature searches was conducted on June 2, 2020, which was followed by a review and discussions. The reviewers (XJ, HM, and YG) refined and agreed with the search strategies and searched PubMed, the Association for Computing Machinery (ACM) Digital Library, and the Nursing & Allied Health Database (NAHD) for literature, using the search strategies mentioned below. A final search was conducted on January 5, 2022, in the 3 literature databases as an update.

    For PubMed, the following search was conducted: (clinical decision support systems[MeSH Terms]) AND (ontolog*[Title/abstract] OR rule*[Title/abstract]). For ACM Digital Library, the following search was conducted within the scope of the ACM Guide to Computing Literature: [[Publication Title: “clinical decision support*”] OR [Publication Title: cds*]] AND [[Publication Title: ontolog*] OR [Abstract: ontolog*] OR [Publication Title: rule*] OR [Abstract: rule*]]. For the NAHD, the following search was limited to peer-reviewed publications: mesh(clinical decision support) AND (ti(ontology) OR ti(ontologies) OR ab(ontology) OR ab(ontologies) OR ti(rule) OR ti(rules) OR ab(rule) OR ab(rules)).

    ‎
    Figure 1. General workflow of the systematic literature review.
    View this figure

    Inclusion and Exclusion Criteria

    The inclusion criteria were as follows: text was written in English; full-text publication was available; ontologies were designed to be implemented or were already implemented in CDSSs, particularly related to CDSS rules; content included the granularity of CDSS rules; ontologies were designed to be integrated or were already integrated with health information systems (eg, EHRs), either in a production system or a prototype, with at least one architecture diagram, applied in clinical domains or designed for clinical domains to support health care providers; the publication was peer-reviewed; and details on the integration of CDSSs and EHRs were present for evaluation studies.

    The exclusion criteria were as follows: only CDSS rules were included, regardless of the stage of the CDSS rule lifecycle (ie, development, identification, refinement, validation, evaluation, or implementation) or there was no mention of integration or ontologies; only ontologies were developed, evaluated, and validated, or there was no mention of integration or a CDSS; the system was designed without mentioning the granularity of CDSS rules or ontologies; and nonclinical decisions, such as administrative or management decisions (eg, supply chain management), were described.

    General Workflow for Screening Papers

    The first 100 papers were screened by all 3 authors (XJ, HM, and YG) independently. The first 100 retrieved papers were initially screened by 1 author (XJ) to draft initial inclusion and exclusion criteria. The inclusion and exclusion criteria were refined and adjusted by 2 authors (HM and YG) during the iterative screening, review, analysis, and discussions. Further, 2 authors (HM and YG) replicated the screening, and all 3 authors discussed and validated the results. The rest of the papers were then screened by at least 2 authors (XJ and HM, or XJ and YG) independently to determine inclusion. Disagreements were discussed and resolved via iterative rounds of group meetings.

    The screening and manual review processes were conducted independently and approved by at least 2 authors. The literature was first screened based on titles, abstracts, and full-text publications when needed. The papers that were included were then manually coded to provide more content analysis and synthesized evidence. The final results were shared among all the authors. All disagreements were settled through group discussions.

    Reviewing, Coding, Analyzing, and Synthesizing Processes

    We followed grounded theory during the reviewing and manual coding of the included publications. One author (XJ) randomly selected 10 papers from the included 81 papers to start the coding (annotating) based on the focus of this literature review. ATLAS.ti 9 (desktop and web versions; ATLAS.ti Scientific Software Development GmbH), a qualitative data analytic tool, was used for coding. The coding results were discussed by 3 authors (XJ, HM, and YG). The discussion results formed the first draft of codes and code groups (Multimedia Appendix 1), that is, data items. Three coders (XJ, HM, and YG) then reviewed and coded the first 40 of the included papers using the initial principles and code groups, and added new codes and code groups when needed. Then, a second set of meetings was used to obtain consensus on updated principles and code groups. Refined codes and code groups were used to code the remaining papers. Every paper was coded by at least 2 coders independently. The coding results were then compared, and any discrepancies were resolved by group discussions. The code groups and codes were revised, consolidated, and updated during each discussion. Multimedia Appendix 2 presents the refined code groups and examples. Data items emerged during the review and were refined via discussions instead of predefinition before reviewing. Multimedia Appendix 3 lists all included papers.

    After coding, the literature was analyzed and synthesized with a focus on several aspects, including CDSS application domains, CDSS mechanisms used in clinical settings, CDSS rule formats, authoring, management, and the roles of ontologies. The 3 authors worked together in an iterative process of analysis and synthetization. After obtaining consensus among all 3 authors, the results were then shared and discussed among all authors. Any concerns, confusions, or disagreements among the authors were resolved through iterative discussions. We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 checklist [50] for reporting the systematic review with all relevant items (Multimedia Appendix 4 and Multimedia Appendix 5).


    Overview

    By January 5, 2022, literature searches retrieved 1235 publications from 3 sources. After removing duplicates and examining according to the inclusion and exclusion criteria, 81 publications (Multimedia Appendix 3) were included in the final review and analysis [26,27,29,31-33,51-125]. Figure 2 depicts the literature search, screening, selection flow, and results. Figure 3 summarizes the main components covered by the literature review and the summary findings, and serves as an initial knowledge framework on CDSSs, CDSS rules, and ontology applications in CDSSs.

    ‎
    Figure 2. Flowchart of the literature search, screening, and selection. ACM: Association for Computing Machinery.
    View this figure
    ‎
    Figure 3. Initial knowledge framework on clinical decision support systems (CDSSs), CDSS rules, and ontology applications in CDSSs. ICD-10: International Statistical Classification of Diseases and Related Health Problems, 10th Revision; LOINC: Logical Observation Identifiers Names and Codes; MLM: medical logic module; OWL: Web Ontology Language; SNOMED CT: Systematized Nomenclature of Medicine-Clinical Terms; SWRL: Semantic Web Rule Language; UMLS: Unified Medical Language System; XML: Extensive Markup Language.
    View this figure

    The majority of the publications (73/81, 90%) included in the review were from PubMed, a dominant source. After removing duplicates, the ACM Library added 8 new publications. After cleaning, discussion, and consolidation, 30 code groups and 221 final codes were used in ATLAS.ti (Multimedia Appendix 2). These codes and code groups guided our analysis and synthesis of the results. Multimedia Appendix 6 shows a word cloud image generated by ATLAS.ti that reflects the codes coded in the publications included.

    PRISMA 2020 is designed to guide the reporting of outcome-oriented studies. Our systematic literature review focused on the design, development, and implementation of CDSSs, particularly related to CDSS rules and ontologies. Therefore, effect measures or certainty assessments were irrelevant items. We reported 19 categorical items (out of 27 categorical items, 26 items out of 42 items; Multimedia Appendix 4) for the full-text papers and 10 for the abstracts (out of 12 items; Multimedia Appendix 5).

    Results Related to CDSS Characteristics

    Over one-third (29/81, 36%) of CDSSs were designed and used for chronic condition management, prediction, or risk assessment, including but not limited to type 1 and 2 diabetes, hypertension, and asthma. Medication prescriptions (13/81, 16%), such as medication ordering, detection of adverse drug events, drug-drug interactions, and cancer care (8/81, 10%), were also significant application domains. Multimedia Appendix 7 illustrates the clinical domains of CDSSs within the included publications. Most CDSSs were designed for health care providers, but only 11% (9/81) were intended for patients. Most CDSSs provided recommendations, suggestions, alerts, or reminders. Among all the items in our comparison (Multimedia Appendix 8), EHR evaluation studies within the operational systems or prototypes exhibited the least complete information. Evaluations of CDSSs have been listed in multiple columns in Multimedia Appendix 8. Some CDSSs were implemented in production systems (31/81, 38%), whereas others were implemented in prototypes (30/81, 37%), which included experimental systems. Multimedia Appendix 8 summarizes the key features of CDSSs identified in the publications. In all tables, we adopted the original terms used in the corresponding papers. Some papers, for example, referred to “physicians” as CDSS users, whereas others referred to “clinicians” as CDSS users.

    Results Related to CDSS Rules

    Most CDSS rules were written in Web Ontology Language (OWL; 11/81, 14%), Extensive Markup Language (XML; 10/81, 12%), SWRL (9/81, 11%), Jena rules (5/81, 6%), and medical logic module (MLM; 3/81, 4%). Moreover, 2 publications [117,119] used N3 Language and 2 [90,117] used Natural Rule Language (NRL). Multimedia Appendix 9 presents 54 publications with more details on the CDSS rules, that is, publications that can fill out 3 or more cells (except for authors and publication year).

    The most significant CDSS rule source is from clinical practice guidelines (36/81, 44%). Other sources of CDSS rules included domain expert input, publications (eg, textbooks and papers), multimedia sources, and internet resources. Data mining results were involved in CDSS rule sources [67,73]. CDSS rule authoring and editing tools were not routinely specified in the publications. Protégé [115] was the most prevalent tool to edit and author CDSS rules. Several publications also described developing authoring and editing tools [57,65,91].

    There was a lack of technical details regarding rule engines, among which Jena (6/81, 7%), inference engine (6/81, 7%), Jess (4/81, 5%), JBoss (3/81, 4%), guideline engine (3/81, 4%), Drools (2/81, 3%), and Bayes (2/81, 3%) were frequently mentioned. Multimedia Appendix 9 summarizes how the CDSS rule (operation) works in a simplified manner. Many publications did not specify the working mechanism of CDSS rules within the EHR, electronic medical record (EMR), or hospital information system (HIS) context.

    The majority of the publications did not appear to be focused on interoperability. Few papers that discussed interoperability (Multimedia Appendix 9) used HL7 CDA (Health Level 7, Clinical Document Architecture) or HL7 FHIR (Health Level 7, Fast Healthcare Interoperability Resources) standards. However, it is worth noting that such HL7 measures were not specifically designed for CDSS rules but rather for CDSS input and output.

    Furthermore, some publications lacked necessary information for explaining the mechanisms of the systems, which can be critical barriers to reproducibility. Some publications lacked critical information, such as CDSS architecture diagrams; CDSS rule engines; CDSS rule languages; backend management methods for CDSS rules; and integration mechanisms among CDSS rules, ontologies, and EHR, EMR, or HIS systems.

    Results Related to Ontologies

    In the included publications, ontologies were primarily used as knowledge sources for CDSSs (32/81, 40%) to facilitate classification (7/81, 9%), reasoning, and inference (6/81, 7%; eg, identification recommendations or relationships). Moreover, ontologies were used to specify CDSS rules (12/81, 15%) or to provide general knowledge for the EMR or EHR systems. These 2 applications overlapped in some cases (19/81, 24%; ie, the ontologies were used to provide specified CDSS rules and general knowledge).

    In the included publications, the terms “reasoner” and “rule engines” were used interchangeably. Reasoner, in our opinion, refers to the inference for a consistency check or classification for an ontology. A reasoner can be part of an ontology tool or can be external. For CDSSs, a rule engine is the mechanism that generates or provides recommendations by incorporating a patient’s data, contextual information, and medical knowledge (typically from an ontology or knowledge base). However, we kept the authors’ choice of terms in tables without modification. Among the included publications, the most common reasoners were Pellet (11/81, 14%), Jena (4/81, 5%), OWL reasoner (3/81, 4%), Jess (2/81, 3%), and the Euler/EYE inference engine (2/81, 3%).

    The content and code systems used to represent the content should be included as ontology sources. The content could come from a popular textbook or a clinical practice guideline. The content can be coded in a specific code system, such as SNOMED CT (Systematized Nomenclature of Medicine-Clinical Terms). Multimedia Appendix 10 includes code systems that served as ontology sources. The most often used coding systems among the included publications were SNOMED CT (9/81, 11%), the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10; 4/81, 5%), Unified Medical Language System (UMLS; 4/81, 5%), Logical Observation Identifiers Names and Codes (LOINC; 3/81, 4%), and RxNorm (2/81, 3%).

    Incomplete information for ontology validation is a common issue shown in the literature. Approximately 20 publications mentioned some validation, including validation or evaluation by domain experts (20/81, 25%). Some ontologies were authored by domain experts [55,63]. Multimedia Appendix 10 provides more information on the roles of ontologies in publications, including publications with 3 or more cells (except for authors and publication year; n=36).


    Summary of the Results

    Although ontologies contribute to the content of CDSS rules and have the potential to facilitate interoperable CDSS rules, our systematic review showed that reusing and sharing of CDSS rules have not been achieved. CDSSs have a wide range of clinical application domains, primarily for health care providers, such as chronic condition management, medication ordering, and cancer care. CDSS rules are primarily based on clinical practice guidelines.

    Although reusing and sharing CDSS artifacts are well-recognized challenges [1,109], reusability, customization, and shareability of CDSS rules are not yet a common focus, even in publications focusing on CDSS rule editing [43,126,127]. These are important topics to cover in a literature review. Marco-Ruiz et al [109] demonstrated how to use CDSS artifacts in the Linked Data framework [128] by leveraging Semantic Web technologies, particularly ontology. However, that work was at a higher level, describing concepts without tangible tools implemented in clinical practice. To fill this gap, one approach is to build an upper-level CDSS ontology [129] to encourage the reuse of CDSS rules and demonstrate the potential of ontologies. Our effort is in alignment with their vision, as well as other efforts in reusing and sharing CDSS artifacts [1,109].

    Ontologies were not at the center of any early examples of CDSSs [6-12]. An early demonstration of using medical terminology in CDSSs was the adoption of Current Medical Information and Terminology (CMIT) in a diagnostic engine [130,131]. Even under our “loose use of ontology” during our systematic literature search, there was no case in which ontology played a central role in sharing CDSS rules, particularly for rule management and maintenance.

    Over the years, CDSSs have been successfully applied in clinical care. Unfortunately, CDSS rules are not yet portable. Making CDSS rules more portable is therefore significant work that could be leveraged by ontologies, and our systematic literature review brings us one step closer to that goal. Marco-Ruiz et al also conducted a very relevant systematic literature review. However, their focus was on the interoperability mechanisms used in CDSSs [132,133]. According to the results of their systematic literature review, 32% of the included papers used ontologies and 46% used standard terminologies. The findings related to ontologies are similar [132,133] to those of our paper. However, we presented a more detailed and thorough analysis of these technologies used in CDSS rules. Nevertheless, both papers concluded that complete CDSS interoperability is not a reality. Thus, additional efforts are required to achieve interoperable and reusable CDSS artifacts, such as CDSS rules.

    Interpretation of the Results

    Rule engines, which execute rules, patient data, and context information to produce a result, such as an alert or a recommendation, are critical components of CDSSs [1]. Jess, a rule engine and development environment in Java [134], was frequently mentioned in the included publications as a tool for developing rule-based CDSSs. SWRL rules can be converted to Jess rules in the popular tool Protégé, using a plug-in application programming interface (API) SWRLJessTab. Jess rules can be used by the Jess rule engine, which is widely used in rule-based expert systems [134]. In addition to Jess, Jena and Drools were used frequently in the publications included. Jena is a Java API that supports rule-based inference and makes use of resource description framework (RDF) graphs [135]. Jena.java API is a popular framework for managing RDF/OWL descriptions and can handle OWL models [96]. Drools is a business rule management system that includes a rule engine [136]. Drools also has the SWRL API that supports SWRL and Semantic Query-Enhanced Web Rule Language (SQWRL). SWRL can be queried by SQWRL.

    Reasoning via a reasoner is a critical characteristic of many ontologies, even though the current reasoning is still in first-degree logic. Reasoning can be used for the following 3 main functions: consistency check, classification, and realization [137]. Several publications specified the classification roles of the ontologies and reasoners (Multimedia Appendix 10). The Manchester University OWL group has curated an updated list of OWL reasoners [137]. Parsia et al [137] compiled and compared the current OWL reasoners and their performances via the competition report. Both Pellet and Jena are popular reasoners (Multimedia Appendix 10), and other reasoners include FaCT++ [98], Z3 Solver reasoner [105], Euler/EYE inference engine [117,119], OWL Horst [109], and OWL Cerebra [63] among the included publications. Among these reasoners, Pellet [138] is Java based, and it can work on SWRL rules and ontologies written in OWL2. SWRL was initially designed as a rule language for Semantic Web technologies [139]. A user needs the rule language and an editor (eg, Protégé SWRL tab) to write, revise, and query the rules. SWRL can be queried by SQWRL (a query language for OWL) or SPARQL (SPARQL Protocol and RDF query language). Reasoners can then be used to conduct reasoning based on the rules and facts defined in the ontology or knowledge base. Protégé-OWL [140] provides an editor for SWRL rules. Protégé SWRL editor is another example.

    This review has demonstrated unique insights about CDSS rules, ontologies, and ontology applications, particularly in CDSS rule management and maintenance, and has presented several distinct characteristics that complement the existing literature. An earlier review [40] focused on clinical decision-making in forming ontologies to support complex cognitive processes and reasoning processes comparing evaluation metrics but did not cover the implementation of EHR, EMR, or HIS systems and the mechanisms of these characteristics.

    Significance of the Work

    Our systematic review demonstrated the state-of-the-art applications of ontologies in CDSS rules. These applications have a lot of potential for reusing and sharing CDSS artifacts. However, none of the existing papers elaborated or demonstrated how ontologies enable portable CDSS rules. Although some authors recognized this benefit [1,43,109], none have conducted a systematic review. Our literature review thoroughly examined the topic, outlined the current frontlines on CDSS rules and ontology uses in CDSSs, established the knowledge framework, and compiled a comprehensive collection of relevant publications that can inform future efforts to design or improve CDSSs. This systematic review focused on the mechanisms of CDSSs in clinical practices or prototypes, CDSS rules, and ontology roles in CDSSs. The detailed information provided in each included publication (Multimedia Appendix 8, Multimedia Appendix 9, and Multimedia Appendix 10) about the reasoners, rule engines, ontologies, and CDSS rule formats used provided valuable references for designing or improving systems. The side-by-side comparison of publications (Multimedia Appendix 8, Multimedia Appendix 9, and Multimedia Appendix 10) also provided structured guidance for preparing future designs and publications or teaching references on the topics in tangible ways.

    Missing Information in the Publications and Our Recommendations

    Inconsistent or missing information about CDSS rule languages, CDSS rule engines, and CDSS evaluation details was identified. In CDSS evaluation, there was commonly no information about how the evaluation was conducted or who performed the evaluation. There were also inconsistencies in technical details related to ontology purposes, reasoners, connection mechanisms, or communications between CDSSs and EHR, EMR, or HIS systems. Inconsistent or missing information hampered reproducibility and further improvement of published work. We are obviously not the only group that has identified missing critical information as a problem in technical papers on similar topics [141].

    Another missing piece is the evaluation and validation of ontologies or knowledge bases. Only 25% of publications mentioned that domain experts conducted evaluation or validation. A formal assessment or validation is critical to ensure the validity of the results from automated processes for some ontologies (or knowledge bases) derived from other automatic methods (eg, machine learning algorithms). Testing has not been conducted consistently across the publications. Some ontologies were authored by domain experts, which provides greater validity than those involving nondomain experts while constructing ontologies.

    Thus, it is recommended that authors include essential technical details in publications. These technical details include CDSS application domains, intended CDSS users, CDSS notification types, CDSS evaluations (what, how, and by whom), CDSS rule sources, CDSS rule languages, CDSS rule engines, CDSS operation mechanisms, ontology use purposes, ontology sources (both content and code systems), ontology validation, reasoners, and connection or communication mechanisms between CDSSs and EMR, EHR, or HIS systems. Authors are highly encouraged to include such details to help readers reference, compare, and increase the reproducibility of the reported work.

    Limitations

    Our review has limitations. Non-English publications or full-text unavailable publications were not included. Publications that focused only on CDSS rules [43,126,127] were also excluded. Moreover, publications without specifying an ontology component were excluded, although such publications had a similar focus to one aspect of our systematic review. We also noticed that most of the publications on CDSS rule authoring and managing tools were from Partners HealthCare/Harvard Medical School. The strengths of Partners HealthCare/Harvard Medical School were shown. On the other hand, a lack of broad adoption, implementation, or publication of such topics was shown.

    When “CDSS” is not specified as a keyword, the search results may exclude publications. For example, our 2 previous papers [142,143] were not found via the search strategy because “CDSS” was not used as a keyword, although the content was undoubtedly within the scope of this review. This challenge is common to how our current literature databases are organized and how we conduct a literature search. Even with MeSH (Medical Subject Headings; the controlled vocabulary for PubMed), publications can still be missed without using commonly recognized keywords. This challenge could be minimized and mitigated by carefully developing an exhaustive list of keywords to maximize the possibilities found during a literature search in the future.

    Conclusions

    The reuse, management, and maintenance of CDSS rules are critical yet challenging for their clinical application. Although ontologies have been used to contribute to the content of CDSS rules, they have not been used to facilitate CDSS rule reuse and sharing. Building a CDSS ontology, which could be the first tangible step, requires bridging high-level visions and operational efforts. Semantic interoperability remains a major challenge that must be overcome to achieve reuse of CDSS artifacts, including CDSS rules. The realization of semantic interoperability will not only allow for the reuse of CDSS artifacts, which are resource intensive to develop and maintain, but also provide practical insights to achieve interoperable patient records. This has been a long-lost aspect, and health care providers will be able to access patients’ complete records to provide safer and higher quality care every time to every patient. We believe that making CDSS rules interoperable can provide insightful guidance for interoperable patient records.

    Incomplete technical details on CDSS rules and ontologies presented in publications should be addressed in future publications by including more detailed information about architectural diagrams; the mechanisms of connection among ontologies, CDSS rules, and EHR, EMR, or HIS systems; CDSS rule languages; reasoners; rule engines; the validation or authorization of ontologies and CDSS rules; the purposes of ontologies; ontology sources; and the management and maintenance of CDSS rules. Such information can help researchers to optimize design and development while also increasing reproducibility. Finally, the knowledge framework and the summarization of included publications are expected to guide future CDSS improvements and innovations, CDSS rules, and the integration and communication of CDSSs with EHR, EMR, or HIS systems.

    Acknowledgments

    The authors thank professors James J Cimino, Dean F Sittig, and Adam Wright for their valuable feedback and suggestions on the manuscript. This work was supported by the National Institute of General Medical Sciences of the National Institutes of Health (under award number R01GM138589 and partially under award number P20GM121342). The content is solely the authors’ responsibility and does not necessarily represent the official views of the National Institutes of Health.

    Data Availability

    A request for additional data in addition to the results and appendices of this manuscript can be made to the corresponding author, and the final decision on data release will be made on a case-by-case basis, as appropriate.

    Authors' Contributions

    XJ, HM, and YG designed the study, conducted the review and coding, drafted the first version, analyzed and interpreted the results, and revised the manuscript significantly. PB, DR, TL, CN, AF, LR, NH, and RG participated in the design of the study, analyzed and interpreted the results, and revised the manuscript significantly.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Initial draft of codes and code groups used in reviewing/coding.

    PDF File (Adobe PDF File), 195 KB
    ‎
    Multimedia Appendix 2

    Refined version of codes and code groups used in reviewing/coding.

    PDF File (Adobe PDF File), 215 KB
    ‎
    Multimedia Appendix 3

    List of papers included in this systematic literature review (n=81).

    PDF File (Adobe PDF File), 222 KB
    ‎
    Multimedia Appendix 4

    PRISMA 2020 item checklist reported in our systematic literature review.

    PDF File (Adobe PDF File), 155 KB
    ‎
    Multimedia Appendix 5

    PRISMA 2020 for Abstracts checklist reported in our systematic literature review.

    PDF File (Adobe PDF File), 37 KB
    ‎
    Multimedia Appendix 6

    Word cloud generated from ATLAS.ti based on codes in the included publications .

    PNG File , 353 KB
    ‎
    Multimedia Appendix 7

    Bar chart generated from ATLAS.ti showing the clinical domains of clinical decision support systems in included publications.

    PNG File , 122 KB
    ‎
    Multimedia Appendix 8

    Basic clinical decision support system profiles in included publications (n=81).

    PDF File (Adobe PDF File), 198 KB
    ‎
    Multimedia Appendix 9

    Comparison of clinical decision support system rule characteristics in included publications (n=54).

    PDF File (Adobe PDF File), 182 KB
    ‎
    Multimedia Appendix 10

    Comparison of ontology roles in included publications (n=36).

    PDF File (Adobe PDF File), 151 KB
    1. Greenes RA. Clinical Decision Support: The Road to Broad Adoption. Cambridge, MA: Academic Press; 2014.
    2. Berner ES. Clinical decision support systems: State of the Art. AHRQ. Rockville, MD: Agency for Healthcare Research and Quality; 2009.   URL: https://digital.ahrq.gov/sites/default/files/docs/biblio/09-0069-EF_1.pdf [accessed 2022-12-21]
    3. Sittig DF, Wright A, Meltzer S, Simonaitis L, Evans RS, Nichol WP, et al. Comparison of clinical knowledge management capabilities of commercially-available and leading internally-developed electronic health records. BMC Med Inform Decis Mak 2011 Feb 17;11:13 [FREE Full text] [CrossRef] [Medline]
    4. Sittig DF, Wright A, Simonaitis L, Carpenter JD, Allen GO, Doebbeling BN, et al. The state of the art in clinical knowledge management: an inventory of tools and techniques. Int J Med Inform 2010 Jan;79(1):44-57 [FREE Full text] [CrossRef] [Medline]
    5. Moja L, Polo Friz H, Capobussi M, Kwag K, Banzi R, Ruggiero F, et al. Effectiveness of a Hospital-Based Computerized Decision Support System on Clinician Recommendations and Patient Outcomes: A Randomized Clinical Trial. JAMA Netw Open 2019 Dec 02;2(12):e1917094 [FREE Full text] [CrossRef] [Medline]
    6. Pople HE, Myers JD, Miller RA. DIALOG: a model of diagnostic logic for internal medicine. In: IJCAI'75: Proceedings of the 4th International Joint Conference on Artificial Intelligence - Volume 1. 1975 Presented at: 4th International Joint Conference on Artificial Intelligence; September 3-8, 1975; Tbilisi, Georgia, USSR p. 848-855. [CrossRef]
    7. Miller RA. A history of the INTERNIST-1 and Quick Medical Reference (QMR) computer-assisted diagnosis projects, with lessons learned. Yearb Med Inform 2010:121-136. [Medline]
    8. Miller RA, McNeil MA, Challinor SM, Masarie FE, Myers JD. The INTERNIST-1/QUICK MEDICAL REFERENCE project--status report. West J Med 1986 Dec;145(6):816-822 [FREE Full text] [Medline]
    9. Miller RA, Pople HE, Myers JD. Internist-1, an experimental computer-based diagnostic consultant for general internal medicine. N Engl J Med 1982 Aug 19;307(8):468-476. [CrossRef] [Medline]
    10. Bergeron B. Iliad: a diagnostic consultant and patient simulator. MD Comput 1991;8(1):46-53. [Medline]
    11. Warner HR. Iliad: diagnostic tools for general medicine. Interview by Bill W. Childs. Healthc Inform 1990 Apr;7(4):38. [Medline]
    12. Warner HR, Bouhaddou O. Innovation review: Iliad--a medical diagnostic support program. Top Health Inf Manage 1994 May;14(4):51-58. [Medline]
    13. Lobach D, Sanders G, Bright T, Wong A, Dhurjati R, Bristow E, et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evid Rep Technol Assess (Full Rep) 2012 Apr(203):1-784. [Medline]
    14. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005 Apr 02;330(7494):765 [FREE Full text] [CrossRef] [Medline]
    15. O'Connor PJ, Sperl-Hillen JM, Rush WA, Johnson PE, Amundson GH, Asche SE, et al. Impact of electronic health record clinical decision support on diabetes care: a randomized trial. Ann Fam Med 2011;9(1):12-21 [FREE Full text] [CrossRef] [Medline]
    16. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med 1976 Dec 09;295(24):1351-1355. [CrossRef] [Medline]
    17. Middleton B, Sittig DF, Wright A. Clinical Decision Support: a 25 Year Retrospective and a 25 Year Vision. Yearb Med Inform 2016 Aug 02;Suppl 1(Suppl 1):S103-S116 [FREE Full text] [CrossRef] [Medline]
    18. Wright A, Ash JS, Aaron S, Ai A, Hickman TT, Wiesen JF, et al. Best practices for preventing malfunctions in rule-based clinical decision support alerts and reminders: Results of a Delphi study. Int J Med Inform 2018 Oct;118:78-85 [FREE Full text] [CrossRef] [Medline]
    19. Horsky J, Schiff GD, Johnston D, Mercincavage L, Bell D, Middleton B. Interface design principles for usable decision support: a targeted review of best practices for clinical prescribing interventions. J Biomed Inform 2012 Dec;45(6):1202-1216 [FREE Full text] [CrossRef] [Medline]
    20. Wu R, Peters W, Morgan MW. The next generation of clinical decision support: linking evidence to best practice. J Healthc Inf Manag 2002;16(4):50-55. [Medline]
    21. Fox J, Gutenstein M, Khan O, South M, Thomson R. OpenClinical.net: A platform for creating and sharing knowledge and promoting best practice in healthcare. Computers in Industry 2015 Jan;66:63-72. [CrossRef]
    22. Teich JM, Osheroff JA, Pifer EA, Sittig DF, Jenders RA, CDS Expert Review Panel. Clinical decision support in electronic prescribing: recommendations and an action plan: report of the joint clinical decision support workgroup. J Am Med Inform Assoc 2005;12(4):365-376 [FREE Full text] [CrossRef] [Medline]
    23. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clinical decision support. J Biomed Inform 2008 Apr;41(2):387-392 [FREE Full text] [CrossRef] [Medline]
    24. Cimino JJ. An integrated approach to computer-based decision support at the point of care. Trans Am Clin Climatol Assoc 2007;118:273-288 [FREE Full text] [Medline]
    25. Jing X, Himawan L, Law T. Availability and usage of clinical decision support systems (CDSSs) in office-based primary care settings in the USA. BMJ Health Care Inform 2019 Dec 08;26(1):e100015 [FREE Full text] [CrossRef] [Medline]
    26. Borbolla D, Otero C, Lobach DF, Kawamoto K, Gomez Saldaño AM, Staccia G, et al. Implementation of a clinical decision support system using a service model: results of a feasibility study. Stud Health Technol Inform 2010;160(Pt 2):816-820. [Medline]
    27. Goldberg HS, Paterno MD, Rocha BH, Schaeffer M, Wright A, Erickson JL, et al. A highly scalable, interoperable clinical decision support service. J Am Med Inform Assoc 2014 Feb 01;21(e1):e55-e62 [FREE Full text] [CrossRef] [Medline]
    28. Miller PL, Frawley SJ, Sayward FG, Yasnoff WA, Duncan L, Fleming DW. IMM/Serve: a rule-based program for childhood immunization. Proc AMIA Annu Fall Symp 1996:184-188 [FREE Full text] [Medline]
    29. Nguyen B, Reese T, Decker S, Malone D, Boyce RD, Beyan O. Implementation of Clinical Decision Support Services to Detect Potential Drug-Drug Interaction Using Clinical Quality Language. Stud Health Technol Inform 2019 Aug 21;264:724-728. [CrossRef] [Medline]
    30. Linder J, Schnipper JL, Volk LA, Tsurikova R, Palchuk M, Olsha-Yehiav M, et al. Clinical decision support to improve antibiotic prescribing for acute respiratory infections: results of a pilot study. AMIA Annu Symp Proc 2007 Oct 11;2007:468-472 [FREE Full text] [Medline]
    31. Ray HN, Boxwala AA, Anantraman V, Ohno-Machado L. Providing context-sensitive decision-support based on WHO guidelines. Proc AMIA Symp 2002:637-641 [FREE Full text] [Medline]
    32. Barth C, Tobman M, Nätscher C, Sussmann H, Horsch A. Fusing a systematic and a case-based repository for medical decision support. Stud Health Technol Inform 2003;95:560-564. [Medline]
    33. Haug PJ, Ferraro JP, Holmen J, Wu X, Mynam K, Ebert M, et al. An ontology-driven, diagnostic modeling system. J Am Med Inform Assoc 2013 Jun 01;20(e1):e102-e110 [FREE Full text] [CrossRef] [Medline]
    34. Del Fiol G, Curtis C, Cimino JJ, Iskander A, Kalluri ASD, Jing X, et al. Disseminating context-specific access to online knowledge resources within electronic health record systems. Stud Health Technol Inform 2013;192:672-676 [FREE Full text] [Medline]
    35. Del Fiol G, Williams MS, Maram N, Rocha RA, Wood GM, Mitchell JA. Integrating genetic information resources with an EHR. AMIA Annu Symp Proc 2006;2006:904 [FREE Full text] [Medline]
    36. Cimino JJ, Overby CL, Devine EB, Hulse NC, Jing X, Maviglia SM, et al. Practical choices for infobutton customization: experience from four sites. AMIA Annu Symp Proc 2013;2013:236-245 [FREE Full text] [Medline]
    37. Del Fiol G, Haug PJ, Cimino JJ, Narus SP, Norlin C, Mitchell JA. Effectiveness of topic-specific infobuttons: a randomized controlled trial. J Am Med Inform Assoc 2008;15(6):752-759 [FREE Full text] [CrossRef] [Medline]
    38. Cimino JJ. Infobuttons: anticipatory passive decision support. AMIA Annu Symp Proc 2008 Nov 06:1203-1204. [Medline]
    39. Patkar V, Acosta D, Davidson T, Jones A, Fox J, Keshtgar M. Using computerised decision support to improve compliance of cancer multidisciplinary meetings with evidence-based guidance. BMJ Open 2012;2(3):e000439 [FREE Full text] [CrossRef] [Medline]
    40. Sutton DR, Fox J. The Syntax and Semantics of the PRO Guideline Modeling Language. J Am Med Inform Assoc 2003 Sep 01;10(5):433-443. [CrossRef]
    41. Del Fiol G, Kohlmann W, Bradshaw RL, Weir CR, Flynn M, Hess R, et al. Standards-Based Clinical Decision Support Platform to Manage Patients Who Meet Guideline-Based Criteria for Genetic Evaluation of Familial Cancer. JCO Clinical Cancer Informatics 2020 Nov(4):1-9. [CrossRef]
    42. Hripcsak G. Writing Arden Syntax Medical Logic Modules. Comput Biol Med 1994 Sep;24(5):331-363. [CrossRef] [Medline]
    43. Zhou L, Karipineni N, Lewis J, Maviglia SM, Fairbanks A, Hongsermeier T, et al. A study of diverse clinical decision support rule authoring environments and requirements for integration. BMC Med Inform Decis Mak 2012 Nov 12;12(1):128 [FREE Full text] [CrossRef] [Medline]
    44. Davies J, Fensel D, van Harmelen F. Towards the Semantic Web: Ontology-driven Knowledge Management. Hoboken, NJ: John Wiley & Sons; 2002.
    45. What is an Ontology? Stanford Knowledge Systems, AI Laboratory.   URL: http://www-ksl.stanford.edu/kst/what-is-an-ontology.html [accessed 2022-12-21]
    46. Musen MA. Dimensions of knowledge sharing and reuse. Comput Biomed Res 1992 Oct;25(5):435-467. [CrossRef] [Medline]
    47. Whetzel PL, Noy NF, Shah NH, Alexander PR, Nyulas C, Tudorache T, et al. BioPortal: enhanced functionality via new Web services from the National Center for Biomedical Ontology to access and use ontologies in software applications. Nucleic Acids Res 2011 Jul;39(Web Server issue):W541-W545 [FREE Full text] [CrossRef] [Medline]
    48. Dissanayake PI, Colicchio TK, Cimino JJ. Using clinical reasoning ontologies to make smarter clinical decision support systems: a systematic review and data synthesis. J Am Med Inform Assoc 2020 Jan 01;27(1):159-174 [FREE Full text] [CrossRef] [Medline]
    49. Samwald M, Stenzhorn H, Dumontier M, Marshall MS, Luciano J, Adlassnig K. Towards an interoperable information infrastructure providing decision support for genomic medicine. Stud Health Technol Inform 2011;169:165-169. [Medline]
    50. Page MJ, McKenzie JE, Bossuyt PM, Boutron I, Hoffmann TC, Mulrow CD, et al. The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. J Clin Epidemiol 2021 Jun;134:178-189 [FREE Full text] [CrossRef] [Medline]
    51. Manjarrés Riesco A, Martı́nez Tomás R, Mira Mira J. A customisable framework for the assessment of therapies in the solution of therapy decision tasks. Artificial Intelligence in Medicine 2000 Jan;18(1):57-82. [CrossRef]
    52. Abidi SS, Manickam S. Transforming XML-based electronic patient records for use in medical case based reasoning systems. Stud Health Technol Inform 2000;77:709-713. [Medline]
    53. De Clercq PA, Blom JA, Hasman A, Korsten HH. GASTON: an architecture for the acquisition and execution of clinical guideline-application tasks. Med Inform Internet Med 2000;25(4):247-263. [CrossRef] [Medline]
    54. Payne TH, Savarino J, Marshall R, Hoey CT. Use of a clinical event monitor to prevent and detect medication errors. Proc AMIA Symp 2000:640-644 [FREE Full text] [Medline]
    55. Achour SL, Dojat M, Rieux C, Bierling P, Lepage E. A UMLS-based knowledge acquisition tool for rule-based clinical decision support system development. J Am Med Inform Assoc 2001;8(4):351-360 [FREE Full text] [CrossRef] [Medline]
    56. Séroussi B, Bouaud J, Dréau H, Falcoff H, Riou C, Joubert M, et al. ASTI: a guideline-based drug-ordering system for primary care. Stud Health Technol Inform 2001;84(Pt 1):528-532. [Medline]
    57. Karadimas HC, Chailloleau C, Hemery F, Simonnet J, Lepage E. Arden/J: an architecture for MLM execution on the Java platform. J Am Med Inform Assoc 2002;9(4):359-368 [FREE Full text] [CrossRef] [Medline]
    58. Das AK, Musen MA. SYNCHRONUS: a reusable software module for temporal integration. Proc AMIA Symp 2002:195-199 [FREE Full text] [Medline]
    59. Poon EG, Wang SJ, Gandhi TK, Bates DW, Kuperman GJ. Design and implementation of a comprehensive outpatient Results Manager. J Biomed Inform 2003;36(1-2):80-91 [FREE Full text] [CrossRef] [Medline]
    60. Liaw S, Sulaiman N, Pearce C, Sims J, Hill K, Grain H, et al. Falls Prevention within the Australian General Practice Data Model: Methodology, Information Model, and Terminology Issues. J Am Med Inform Assoc 2003 Sep 01;10(5):425-432. [CrossRef]
    61. Greenes RA, Sordo M, Zaccagnini D, Meyer M, Kuperman GJ. Design of a standards-based external rules engine for decision support in a variety of application contexts: report of a feasibility study at Partners HealthCare System. Stud Health Technol Inform 2004;107(Pt 1):611-615. [Medline]
    62. Ebrahiminia V, Riou C, Seroussi B, Bouaud J, Dubois S, Falcoff H, et al. Design of a decision support system for chronic diseases coupling generic therapeutic algorithms with guideline-based specific rules. Stud Health Technol Inform 2006;124:483-488. [Medline]
    63. Kashyap V, Morales A, Hongsermeier T. On implementing clinical decision support: achieving scalability and maintainability by combining business rules and ontologies. AMIA Annu Symp Proc 2006;2006:414-418 [FREE Full text] [Medline]
    64. Verlaene K, Joosen W, Verbaeten P. Arriclides: An Architecture Integrating Clinical Decision Support Models. 2007 Presented at: 40th Annual Hawaii International Conference on System Sciences (HICSS'07); January 3-6, 2007; Waikoloa, HI, USA. [CrossRef]
    65. Abidi SR. Ontology-Based Modeling of Breast Cancer Follow-up Clinical Practice Guideline for Providing Clinical Decision Support. 2007 Presented at: Twentieth IEEE International Symposium on Computer-Based Medical Systems (CBMS'07); June 20-22, 2007; Maribor, Slovenia. [CrossRef]
    66. Jannin P, Morandi X. Surgical models for computer-assisted neurosurgery. Neuroimage 2007 Sep 01;37(3):783-791 [FREE Full text] [CrossRef] [Medline]
    67. Stacey M, McGregor C, Tracy M. An architecture for multi-dimensional temporal abstraction and its application to support neonatal intensive care. Annu Int Conf IEEE Eng Med Biol Soc 2007;2007:3752-3756. [CrossRef] [Medline]
    68. Papageorgiou E, Stylios C, Groumpos P. Novel Architecture for supporting medical decision making of different data types based on Fuzzy Cognitive Map Framework. Annu Int Conf IEEE Eng Med Biol Soc 2007;2007:1192-1195. [CrossRef] [Medline]
    69. Cornalba C, Bellazzi R, Bellazzi R. Building a Normative Decision Support System for Clinical and Operational Risk Management in Hemodialysis. IEEE Trans. Inform. Technol. Biomed 2008 Sep;12(5):678-686. [CrossRef]
    70. Farion K, Michalowski W, Wilk S, O’Sullivan D, Rubin S, Weiss D. Clinical Decision Support System for Point of Care Use. Methods Inf Med 2018 Jan 17;48(04):381-390. [CrossRef]
    71. Dao TT, Marin F, Ho Ba Tho MC. Clinical validated Computer-Aided Decision System to the clubfeet deformities. Annu Int Conf IEEE Eng Med Biol Soc 2009;2009:6230-6233. [CrossRef] [Medline]
    72. Zhou Q. A Clinical Decision Support System for Metabolism Synthesis. 2009 Presented at: International Conference on Computational Intelligence and Natural Computing; June 6-7, 2009; Wuhan, China. [CrossRef]
    73. Carenini M, ReMINE Consortium. ReMINE: an ontology-based risk management platform. Stud Health Technol Inform 2009;148:32-42. [Medline]
    74. Lee J, Kim J, Cho I, Kim Y. Integration of workflow and rule engines for clinical decision support services. Stud Health Technol Inform 2010;160(Pt 2):811-815. [Medline]
    75. Ongenae F, Dhaene T, Turck F, Benoit D, Decruyenaere J. Design of a probabilistic ontology-based clinical decision support system for classifying temporal patterns in the ICU: A sepsis case study. 2010 Presented at: 23rd International Symposium on Computer-Based Medical Systems (CBMS); October 12-15, 2010; Bentley, WA, Australia. [CrossRef]
    76. Basilakis J, Lovell NH, Redmond SJ, Celler BG. Design of a decision-support architecture for management of remotely monitored patients. IEEE Trans Inf Technol Biomed 2010 Sep;14(5):1216-1226. [CrossRef] [Medline]
    77. Wilk S, Michalowski W, Farion K, Sayyad Shirabad J. MET3-AE system to support management of pediatric asthma exacerbation in the emergency department. Stud Health Technol Inform 2010;160(Pt 2):841-845. [Medline]
    78. Bouamrane M, Rector A, Hurrell M. Using OWL ontologies for adaptive patient information modelling and preoperative clinical decision support. Knowl Inf Syst 2010 Oct 22;29(2):405-418. [CrossRef]
    79. Dao T, Marin F, Bensahel H, Ho Ba Tho MC. Computer-aided decision system for the clubfeet deformities. Adv Exp Med Biol 2011;696:623-635. [CrossRef] [Medline]
    80. Cao F, Sun X, Wang X, Li B, Li J, Pan Y. Ontology-based knowledge management for personalized adverse drug events detection. Stud Health Technol Inform 2011;169:699-703. [Medline]
    81. Lee CS, Wang MH. A fuzzy expert system for diabetes decision support application. IEEE Trans Syst Man Cybern B Cybern 2011 Feb;41(1):139-153. [CrossRef] [Medline]
    82. Riaño D, Real F, López-Vallverdú JA, Campana F, Ercolani S, Mecocci P, et al. An ontology-based personalization of health-care knowledge to support clinical decisions for chronically ill patients. J Biomed Inform 2012 Jun;45(3):429-446 [FREE Full text] [CrossRef] [Medline]
    83. Bright TJ, Yoko Furuya E, Kuperman GJ, Cimino JJ, Bakken S. Development and evaluation of an ontology for guiding appropriate antibiotic prescribing. J Biomed Inform 2012 Feb;45(1):120-128 [FREE Full text] [CrossRef] [Medline]
    84. Koutkias V, Kilintzis V, Stalidis G, Lazou K, Niès J, Durand-Texte L, et al. Knowledge engineering for adverse drug event prevention: on the design and development of a uniform, contextualized and sustainable knowledge-based framework. J Biomed Inform 2012 Jun;45(3):495-506 [FREE Full text] [CrossRef] [Medline]
    85. Grando A, Farrish S, Boyd C, Boxwala A. Ontological approach for safe and effective polypharmacy prescription. AMIA Annu Symp Proc 2012;2012:291-300 [FREE Full text] [Medline]
    86. Chniti A, Boussadi A, Degoulet P, Albert P, Charlet J. Pharmaceutical validation of medication orders using an OWL Ontology and Business Rules. Stud Health Technol Inform 2012;180:1224-1226. [Medline]
    87. Paterno MD, Goldberg HS, Simonaitis L, Dixon BE, Wright A, Rocha BH, et al. Using a service oriented architecture approach to clinical decision support: performance results from two CDS Consortium demonstrations. AMIA Annu Symp Proc 2012;2012:690-698 [FREE Full text] [Medline]
    88. Yao W, Kumar A. CONFlexFlow: Integrating Flexible clinical pathways into clinical decision support systems using context and rules. Decision Support Systems 2013 May;55(2):499-515. [CrossRef]
    89. Artetxe A, Sanchez E, Toro C, Sanín C, Szczerbicki E, Graña M, et al. Impact of reflexive ontologies in semantic clinical decision support systems. Cybernetics and Systems 2013 Mar;44(2-3):187-203. [CrossRef]
    90. Farkash A, Timm JTE, Waks Z. A model-driven approach to clinical practice guidelines representation and evaluation using standards. Stud Health Technol Inform 2013;192:200-204. [Medline]
    91. Sáez C, Bresó A, Vicente J, Robles M, García-Gómez JM. An HL7-CDA wrapper for facilitating semantic interoperability to rule-based Clinical Decision Support Systems. Comput Methods Programs Biomed 2013 Mar;109(3):239-249. [CrossRef] [Medline]
    92. Corrigan D, Taweel A, Fahey T, Arvanitis T, Delaney B. An ontological treatment of clinical prediction rules implementing the Alvarado score. Stud Health Technol Inform 2013;186:103-107. [Medline]
    93. Shojanoori R, Juric R. Semantic remote patient monitoring system. Telemed J E Health 2013 Feb;19(2):129-136. [CrossRef] [Medline]
    94. Wilk S, Michalowski W, O'Sullivan D, Farion K, Sayyad-Shirabad J, Kuziemsky C, et al. A task-based support architecture for developing point-of-care clinical decision support systems for the emergency department. Methods Inf Med 2013;52(1):18-32 [FREE Full text] [CrossRef] [Medline]
    95. Yılmaz ?, Erdur RC, Türksever M. SAMS--a systems architecture for developing intelligent health information systems. J Med Syst 2013 Dec 7;37(6):9989. [CrossRef] [Medline]
    96. Bau C, Chen R, Huang C. Construction of a clinical decision support system for undergoing surgery based on domain ontology and rules reasoning. Telemed J E Health 2014 May;20(5):460-472 [FREE Full text] [CrossRef] [Medline]
    97. Wang H, Zhou T, Tian L, Qian Y, Li J. Creating hospital-specific customized clinical pathways by applying semantic reasoning to clinical data. J Biomed Inform 2014 Dec;52:354-363 [FREE Full text] [CrossRef] [Medline]
    98. Sesen MB, Peake MD, Banares-Alcantara R, Tse D, Kadir T, Stanley R, et al. Lung Cancer Assistant: a hybrid clinical decision support application for lung cancer care. J R Soc Interface 2014 Sep 06;11(98):20140534 [FREE Full text] [CrossRef] [Medline]
    99. Gallerani M, Pelizzola D, Pivanti M, Boni M, Lamma E, Bellodi E. Appropriateness of Repeated Execution of Laboratory Examinations: A CDSS Approach. 2014 Presented at: IEEE International Conference on Healthcare Informatics; September 15-17, 2014; Verona, Italy. [CrossRef]
    100. Stewart SA, Abidi S, Parker L, Bernstein M, Abidi SSR. Clinical guideline-driven personalized self-management diary for paediatric cancer survivors. Stud Health Technol Inform 2014;205:18-22. [Medline]
    101. Shen Y, Colloc J, Jacquet-Andrieu A, Lei K. Emerging medical informatics with case-based reasoning for aiding clinical decision in multi-agent system. J Biomed Inform 2015 Aug;56:307-317 [FREE Full text] [CrossRef] [Medline]
    102. Delaney BC, Curcin V, Andreasson A, Arvanitis TN, Bastiaens H, Corrigan D, et al. Translational Medicine and Patient Safety in Europe: TRANSFoRm--Architecture for the Learning Health System in Europe. Biomed Res Int 2015;2015:961526 [FREE Full text] [CrossRef] [Medline]
    103. Jafarpour B, Abidi SR, Ahmad AM, Abidi SSR. INITIATE: An Intelligent Adaptive Alert Environment. Stud Health Technol Inform 2015;216:285-289. [Medline]
    104. Robles-Bykbaev V, López-Nores M, Pazos-Arias J, Quisi-Peralta D, García-Duque J. An Ecosystem of Intelligent ICT Tools for Speech-Language Therapy Based on a Formal Knowledge Model. Stud Health Technol Inform 2015;216:50-54. [Medline]
    105. Wilk S, Kezadri-Hamiaz M, Rosu D, Kuziemsky C, Michalowski W, Amyot D, et al. Using Semantic Components to Represent Dynamics of an Interdisciplinary Healthcare Team in a Multi-Agent Decision Support System. J Med Syst 2016 Feb 21;40(2):42. [CrossRef] [Medline]
    106. Abidi SR, Cox J, Abusharekh A, Hashemian N, Abidi SSR. A Digital Health System to Assist Family Physicians to Safely Prescribe NOAC Medications. Stud Health Technol Inform 2016;228:519-523. [Medline]
    107. Zhang Y, Tian Y, Zhou T, Araki K, Li J. Integrating HL7 RIM and ontology for unified knowledge and data representation in clinical decision support systems. Comput Methods Programs Biomed 2016 Jan;123:94-108. [CrossRef] [Medline]
    108. Goldberg HS, Paterno MD, Grundmeier RW, Rocha BH, Hoffman JM, Tham E, et al. Use of a remote clinical decision support service for a multicenter trial to implement prediction rules for children with minor blunt head trauma. Int J Med Inform 2016 Mar;87:101-110. [CrossRef] [Medline]
    109. Marco-Ruiz L, Pedrinaci C, Maldonado JA, Panziera L, Chen R, Bellika JG. Publication, discovery and interoperability of Clinical Decision Support Systems: A Linked Data approach. J Biomed Inform 2016 Aug;62:243-264 [FREE Full text] [CrossRef] [Medline]
    110. Zhang Y, Gou L, Zhou T, Lin D, Zheng J, Li Y, et al. An ontology-based approach to patient follow-up assessment for continuous and personalized chronic disease management. J Biomed Inform 2017 Aug;72:45-59 [FREE Full text] [CrossRef] [Medline]
    111. Shang Y, Wang Y, Gou L, Wu C, Zhou T, Li J. Development of a Service-Oriented Sharable Clinical Decision Support System Based on Ontology for Chronic Disease. Stud Health Technol Inform 2017;245:1153-1157. [Medline]
    112. Chen R, Jiang HQ, Huang C, Bau C. Clinical Decision Support System for Diabetes Based on Ontology Reasoning and TOPSIS Analysis. J Healthc Eng 2017;2017:4307508 [FREE Full text] [CrossRef] [Medline]
    113. Kopanitsa G. Integration of Hospital Information and Clinical Decision Support Systems to Enable the Reuse of Electronic Health Record Data. Methods Inf Med 2018 Jan 24;56(03):238-247. [CrossRef]
    114. Abidi S. A Knowledge-Modeling Approach to Integrate Multiple Clinical Practice Guidelines to Provide Evidence-Based Clinical Decision Support for Managing Comorbid Conditions. J Med Syst 2017 Oct 26;41(12):193. [CrossRef] [Medline]
    115. Shen Y, Yuan K, Chen D, Colloc J, Yang M, Li Y, et al. An ontology-driven clinical decision support system (IDDAP) for infectious disease diagnosis and antibiotic prescription. Artif Intell Med 2018 Mar;86:20-32. [CrossRef] [Medline]
    116. Nakawala H, Ferrigno G, De Momi E. Development of an intelligent surgical training system for Thoracentesis. Artif Intell Med 2018 Jan;84:50-63 [FREE Full text] [CrossRef] [Medline]
    117. Séroussi B, Guézennec G, Lamy J, Muro N, Larburu N, Sekar BD, et al. Reconciliation of multiple guidelines for decision support: a case study on the multidisciplinary management of breast cancer within the DESIREE project. AMIA Annu Symp Proc 2017;2017:1527-1536 [FREE Full text] [Medline]
    118. Winter A, Stäubert S, Ammon D, Aiche S, Beyan O, Bischoff V, et al. Smart Medical Information Technology for Healthcare (SMITH). Methods Inf Med 2018 Jul;57(S 01):e92-e105 [FREE Full text] [CrossRef] [Medline]
    119. Seroussi B, Lamy J, Muro N, Larburu N, Sekar BD, Guézennec G, et al. Implementing Guideline-Based, Experience-Based, and Case-Based Approaches to Enrich Decision Support for the Management of Breast Cancer Patients in the DESIREE Project. Stud Health Technol Inform 2018;255:190-194. [Medline]
    120. Jin W, Kim DH. Design and Implementation of e-Health System Based on Semantic Sensor Network Using IETF YANG. Sensors (Basel) 2018 Feb 20;18(2):629 [FREE Full text] [CrossRef] [Medline]
    121. Román-Villarán E, Pérez-Leon FP, Escobar-Rodriguez GA, Martínez-García A, Álvarez-Romero C, Parra-Calderón CL. An Ontology-Based Personalized Decision Support System for Use in the Complex Chronically Ill Patient. Stud Health Technol Inform 2019 Aug 21;264:758-762. [CrossRef] [Medline]
    122. Semenov I, Osenev R, Gerasimov S, Kopanitsa G, Denisov D, Andreychuk Y. Experience in Developing an FHIR Medical Data Management Platform to Provide Clinical Decision Support. Int J Environ Res Public Health 2019 Dec 20;17(1):73 [FREE Full text] [CrossRef] [Medline]
    123. Jafarpour B, Raza Abidi S, Van Woensel W, Raza Abidi SS. Execution-time integration of clinical practice guidelines to provide decision support for comorbid conditions. Artif Intell Med 2019 Mar;94:117-137. [CrossRef] [Medline]
    124. El-Sappagh S, Ali F, Hendawi A, Jang J, Kwak K. A mobile health monitoring-and-treatment system based on integration of the SSN sensor ontology and the HL7 FHIR standard. BMC Med Inform Decis Mak 2019 May 10;19(1):97 [FREE Full text] [CrossRef] [Medline]
    125. Maldonado JA, Marcos M, Fernández-Breis JT, Giménez-Solano VM, Legaz-García MDC, Martínez-Salvador B. CLIN-IK-LINKS: A platform for the design and execution of clinical data transformation and reasoning workflows. Comput Methods Programs Biomed 2020 Dec;197:105616. [CrossRef] [Medline]
    126. Regier R, Gurjar R, Rocha RA. A clinical rule editor in an electronic medical record setting: development, design, and implementation. AMIA Annu Symp Proc 2009 Nov 14;2009:537-541 [FREE Full text] [Medline]
    127. Sordo M, Rocha BH, Morales AA, Maviglia SM, Oglio ED, Fairbanks A, et al. Modeling decision support rule interactions in a clinical setting. Stud Health Technol Inform 2013;192:908-912. [Medline]
    128. Bizer C, Heath T, Berners-Lee T. Linked Data - The Story So Far. International Journal on Semantic Web and Information Systems 2009;5(3):1-22 [FREE Full text] [CrossRef]
    129. Jing X, Min H, Gong Y. A clinical decision support system (CDSS) ontology to facilitate portable vaccination CDSS rules: preliminary results. 2021 Presented at: AMIA 2021 Annual Symposium; October 30-November 3, 2021; San Diego, CA.
    130. Collen MF. Origins of medical informatics. West J Med 1986 Dec;145(6):778-785 [FREE Full text] [Medline]
    131. McCray AT. The Theoretical Basis of Medical Information Science. (Reflections on Marsden S. Blois' paper on the proper use of man and machines). Yearb Med Inform 1999(1):280-282. [Medline]
    132. Marco-Ruiz L, Budrionis A, Yigzaw KYY, Bellika JG. Interoperability Mechanisms of Clinical Decision Support Systems: A Systematic Review. In: Proceedings of the 14th Scandinavian Conference on Health Informatics. 2016 Presented at: 14th Scandinavian Conference on Health Informatics; April 6-7, 2016; Gothenburg, Sweden p. 13-21.
    133. Marco-Ruiz L, Bellika JG. Semantic Interoperability in Clinical Decision Support Systems: A Systematic Review. Stud Health Technol Inform 2015;216:958. [Medline]
    134. Jess, the Rule Engine for the Java Platform. Sandia National Laboratories.   URL: https://en.wikipedia.org/wiki/Jess_(programming_language) [accessed 2022-12-21]
    135. Apache Jena. The Apache Software Foundation.   URL: https://jena.apache.org/ [accessed 2022-12-21]
    136. Drools.   URL: https://www.drools.org/ [accessed 2022-12-21]
    137. Parsia B, Matentzoglu N, Gonçalves RS, Glimm B, Steigmiller A. The OWL Reasoner Evaluation (ORE) 2015 Competition Report. J Autom Reason 2017;59(4):455-482 [FREE Full text] [CrossRef] [Medline]
    138. Sirin E, Parsia B, Grau B, Kalyanpur A, Katz Y. Pellet: A practical OWL-DL reasoner. Journal of Web Semantics 2007 Jun;5(2):51-53 [FREE Full text] [CrossRef]
    139. O’Connor M, Knublauch H, Tu S. Supporting Rule System Interoperability on the Semantic Web with SWRL. In: Gil Y, Motta E, Benjamins VR, Musen MA, editors. The Semantic Web – ISWC 2005. ISWC 2005. Lecture Notes in Computer Science, vol 3729. Berlin, Heidelberg: Springer; 2005:974-986.
    140. Musen MA, Protégé Team. The Protégé Project: A Look Back and a Look Forward. AI Matters 2015 Jun;1(4):4-12 [FREE Full text] [CrossRef] [Medline]
    141. Moreno-Conde A, Moner D, Cruz WDD, Santos MR, Maldonado JA, Robles M, et al. Clinical information modeling processes for semantic interoperability of electronic health records: systematic review and inductive analysis. J Am Med Inform Assoc 2015 Jul;22(4):925-934 [FREE Full text] [CrossRef] [Medline]
    142. Jing X, Kay S, Marley T, Hardiker NR, Cimino JJ. Incorporating personalized gene sequence variants, molecular genetics knowledge, and health knowledge into an EHR prototype based on the Continuity of Care Record standard. J Biomed Inform 2012 Feb;45(1):82-92 [FREE Full text] [CrossRef] [Medline]
    143. Jing X, Kay S, Marley T, Hardiker NR. Integration of an OWL-DL knowledge base with an EHR prototype and providing customized information. J Med Syst 2014 Sep;38(9):75. [CrossRef] [Medline]

    ‎
    ACM: Association for Computing Machinery
    API: application programming interface
    CDSS: clinical decision support system
    EHR: electronic health record
    EMR: electronic medical record
    HIS: hospital information system
    HL7: Health Level 7
    NAHD: Nursing & Allied Health Database
    OWL: Web Ontology Language
    PRISMA: Preferred Reporting Items for Systematic Reviews and Meta-Analyses
    RDF: resource description framework
    SNOMED CT: Systematized Nomenclature of Medicine-Clinical Terms
    SQWRL: Semantic Query-Enhanced Web Rule Language
    SWRL: Semantic Web Rule Language

    Edited by A Benis; submitted 28.09.22; peer-reviewed by M Bestek, J Schaaf; comments to author 22.10.22; revised version received 16.11.22; accepted 18.12.22; published 19.01.23

    Copyright

    ©Xia Jing, Hua Min, Yang Gong, Paul Biondich, David Robinson, Timothy Law, Christian Nohr, Arild Faxvaag, Lior Rennert, Nina Hubig, Ronald Gimbel. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 19.01.2023.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.