Improving Clinical Decision-Making in Treating Airway Diseases With an Expert System Built Upon the Free AI Tool Google NotebookLM

Introduction

Expert Systems and the Decision-Making Aids

An expert system is a knowledge management tool that integrates domain-specific information with an inference engine to facilitate decision-making. When consulted, it delivers timely and relevant solutions to a problem. This concept originated in the 1950s, with medical diagnostic reasoning being among its earliest applications [1,2]. Gradually, expert systems became popular in the 1970s and 1980s [3]. With the advent of the digital era, the concept of expert systems was revisited by the medical community. For instance, oncology, given its rapid evolution and growth in knowledge over the past 2 decades, has required consultation systems to aid patient management.

Knowledge Management and the Large Language Model

Large language models (LLMs) represent a branch of machine learning that has been under development for several years. Their impact became widely recognized on November 30, 2022, when OpenAI released ChatGPT, a free web-based service [4,5]. This milestone illustrated the potential of LLMs to advance toward general purpose artificial intelligence (AI) by enabling natural language–based dialogue. The continuously expanding medical knowledge has traditionally been managed through structured data storage, indexing, and query mechanisms [6] or probabilistic inference algorithms [7]. However, these conventional clinical decision support systems struggle to incorporate new information and adapt to complex clinical contexts [8]. Unlike these approaches, LLM-driven systems enable interaction in natural language, offering a more user-friendly and adaptable knowledge management tool. Such systems can be easily deployed across health care teams, including physicians and allied health care professionals. In this study, we devised a model to understand how this new LLM-based methodology could affect medical decision-making and routine clinical workflow. The process of our project and our key findings were described as follows.

Decision-Making Aid Construction and Assessment

Constructing an Expert System With a Free AI Tool Based on LLM

The free AI tool Google NotebookLM, powered by the LLM Gemini 2.0 [9], provides a platform for knowledge integration through 3 components: a source input pane, a query pane, and a note-taking pane, which function collaboratively (Figure 1). We used this platform to develop a medical decision support system for airway diseases. Relevant published clinical guidelines were uploaded into the source input pane, including the GINA (Global Initiative for Asthma) guideline [10] and the GOLD (Global Initiative for Chronic Obstructive Lung Disease) guideline [11] (version 2021-2025). Additional standard documents from the Taiwan Society of Pulmonary and Critical Care Medicine [12], such as the Taiwan asthma guidelines and the pulmonary rehabilitation guidelines, were also incorporated. The responses of the system to queries were then recorded.

‎

Feasibility Evaluated by Pulmonologists

Query questions were selected to match the essential requirements of eligible AI tools, such as reproducibility, the ability to develop a treatment plan, and the capacity to handle speculative ideas. They were designed by a pulmonologist with expertise in airway diseases and familiarity with these guidelines, and the appropriateness (focusing on accuracy and completeness) of AI responses was judged (Likert scale 1-5) by 3 pulmonologists independently. Airway diseases were selected as the pilot scenario to evaluate the feasibility of our system because standardized clinical guidelines are widely accepted, and they involve fewer complex and controversial data points than other conditions, such as oncological diseases.

Implementation in the Clinical Workflow

The system was introduced to physicians and allied health care professionals in the emergency department (ED), where patients with airway diseases commonly present for treatment. Feedback was collected through a questionnaire, and the responses were analyzed to assess the system’s effectiveness within the clinical workflow.

The ED setting was chosen because clinicians frequently encounter diverse conditions and require rapid access to specialist knowledge. In this context, an expert system capable of answering clinical queries has the potential to reduce the time burden on patients, frontline clinicians, and the specialists consulted.

The complete workflow of the project is summarized in Figure 2.

‎

AI Responses to Queries

System Behavior Evaluation

Test 1: Reproducibility

The system consistently generated accurate summaries and responses, each supported by appropriate and reliable references. When prompted to regenerate output from the same sources, this system produced slightly varied responses that retained identical meaning, resembling the variability typically observed in an AI chatbot interaction.

Test 2: Knowledge Consolidation

The asthma and chronic obstructive pulmonary disease (COPD) guidelines contained respective lists of differential diagnoses. When instructed to generate a consolidated table of differential diagnoses for airway diseases, including their distinguishing features and relevant diagnostic tests, the system successfully produced one, as shown in Table 1.

Test 3: Temporal Order Recognition

The system demonstrated the ability to recognize temporal updates. For instance, it correctly identified that the “ABCD assessment tool” for COPD was replaced by the “ABE assessment scheme” in the 2023 update of the GOLD COPD guideline [11]. When asked to provide treatment recommendations, the system appropriately referred to the most recent guideline version while disregarding outdated ones, as shown in Textbox 1.

Test 4: Memory Flush

When the COPD guidelines from 2023 onward were deselected from the source input panel, the system could still respond to questions about “ABE assessment” using information retained from previous dialogues, provided those dialogues had not been deleted using the Refresh button (Textbox 2). Interestingly, if the dialogues were deleted and no clues regarding the 2023 update were supplied, the system concluded that “ABE” might be a typographic error for “ABCD” (Textbox 3) or “creatively“ suggested that “B” referred to blood eosinophil count (Textbox 4). This indicates that the AI tool retains dialogue content until it is deliberately erased using the Refresh button.

Test 5: Handling Multiple Languages

Leveraging Google’s multilingual capability, the system could accept a query in Chinese and respond in either English or Chinese. When asked to compare the 2022 Taiwan adult asthma clinical care guidelines, written entirely in Chinese, with the GINA guideline, the system managed to perform the task without difficulty. Language switching did not appear to affect system performance.

Application in Clinical Scenarios

Scenario 1: Requesting Patient Management Suggestions Based on a Simulated Medical Record

As illustrated in Textbox 5, the system generated guideline-based recommendations that were clinically useful.

Scenario 2: Logical Question

The system addressed a logical question regarding the unavailability of pulmonary function testing by providing a structured, entry-by-entry response. It explained the implications separately for COPD and asthma, as shown in Textbox 6.

Scenario 3: Personalized Rehabilitation Planning

The system generated a tailored rehabilitation plan and schedule that took into account an advanced age condition, as shown in Textbox 7.

Scenario 4: Vague Question

In this case, the system transformed the ambiguous concept of “prospect” into practical categories of “future challenges” and “research directions.” This approach provided a meaningful response, as illustrated in Textbox 8.

Scenario 5: Out-of-Scope Questions

The system appropriately declined to answer questions that were beyond the scope of the available sources. It did not fabricate factitious responses, unlike many chatbots.

As a Study Tool for Heuristic Problems

Problem 1: Version Comparison and Trend Analysis

When asked to compare guideline versions across publication years, the system clearly outlined the differences between them. Additionally, when requested to analyze trends in pharmacological use for airway diseases over the past 5 years, it generated a comprehensive summary, as shown in Textbox 9.

Problem 2: Further Reading Suggestions

When asked to recommend additional readings on intractable asthma, the system searched both the source text and the reference lists it contained. However, it did not suggest any external articles or books. Notably, the system interpreted the term “intractable asthma” as both “severe asthma” and “difficult-to-treat asthma” to ensure comprehensive coverage, as shown in Textbox 10.

Problem 3: Speculative Question

We designed a question regarding the interchangeability of questionnaires used for evaluating COPD and asthma. Our system provided a reasonable response, as shown in Textbox 11.

Problem 4: Diagnostic Skill Training Tool

We instructed the system to generate a simulated case with symptoms and pulmonary function values to test our diagnostic skills. During this exercise, we compared our reasoning and diagnosis with the system’s answer to strengthen diagnostic ability, as shown in Textbox 12.

Pulmonologists’ Judgment on AI Responses to Queries

Most of the 3 pulmonologists’ ratings (56/84, 67%) regarding AI responses were above average (Likert scale 4-5), and only 5% (4/84) were below average (Likert scale 1-2). The interrater reliability was moderate for accuracy (intraclass correlation coefficient=0.612; P<.001) and good for completeness (intraclass correlation coefficient=0.773; P<.001) based on a 2-way random-effects model with absolute agreement and single measures options [13]. We also noted that AI responses received more (187/333, 56%) credits from the raters for query questions that required more intelligence (adaptation, deduction, analysis, and planning) to answer, such as scenarios 1 to 3 and problems 1 to 4, than for the other questions. This finding further supports the feasibility of our system.

Responses From the Field Tests in the ED

We collected responses from 20 participants, including 3 physicians, 3 nurse practitioners, 11 ED nurses, and 3 respiratory therapists. Among them, 80% (16/20) expressed interest in the system (Kolmogorov-Smirnov [K-S] D=0.362; P=.008; indicating a deviation from normality). All these participants had prior experience consulting a chatbot. In total, 60% (12/20) of the participants were aware of and ever disturbed by the issue of AI hallucination (K-S D=0.238; P=.18; indicating a normal distribution). All participants in this group thought that guideline-based reasoning provided by the system was essential in clinical practice.

In total, 55% (11/20) of the participants believed that the system could reduce specialty consultation time (K-S D=0.223; P=.24; indicating a normal distribution). Importantly, all participants indicated that the system could enhance their knowledge and diagnostic ability in airway diseases or increase their confidence in explaining conditions to patients.

Discussion

Statistical Findings

Previous studies on LLM-based medical decision support have reported various outcomes [14,15]. These discrepancies may reflect differences in participants’ expertise, the difficulty of test questions, and the dimensions measured in study designs. In our evaluation of consultation time, the K-S test indicated a normal distribution of responses. When analyzing only physicians’ responses, we observed a trend toward favorable outcomes, although this did not reach statistical significance (P=.24) across all participants. Thus, the system shows potential for reducing physicians’ consultation time, but larger and more comprehensive studies are required to confirm its effectiveness.

What We Expect of a Medical AI Consultation Tool

AI Hallucination and AI Ethics in Medicine

Since the widespread adoption of LLM-based chatbots, users have become increasingly concerned about AI hallucination, that is, plausible but unfounded responses generated without a factual basis. Such errors are unacceptable in medical practice. Amiot and Potier [16] recently emphasized that hallucination is particularly difficult to detect in a health care context. Fortunately, the design of the Google NotebookLM platform mitigates this risk by providing answers supported by explicit references to authoritative sources. With such verifiable citations, concerns about errors, biases, ethics, and safety from autonomous AI actions are substantially reduced.

Beyond a Decision-Making Aid

The project aims to provide a clinical decision-making aid for the diagnosis and management of airway diseases and to assess how this AI tool affected medical staff in the ED during the consultation process. As Wang et al [17] noted, an AI system could uncover subtle relationships in medical data that may elude both traditional decision support tools and human experts. During system development, pulmonologists found that the tool not only consolidated their expertise but also offered novel insights and opportunities to explore speculative ideas, as described in test problems 1 and 3. For medical trainees, the system functions as an effective educational tool, providing not only suggestions but also the underlying reasons.

Limitations Inherent to LLMs

Scope of External Knowledge Considered

NotebookLM generates content strictly from user-provided sources. Because general “common sense” information is often absent from “specialized” clinical guidelines, excluding external knowledge can considerably limit the scope of the system’s responses. Moreover, the extent to which external information is incorporated into outputs is undisclosed and therefore unpredictable.

Inability to Pose Differentiating Questions

LLM-based systems generate responses in the form of summaries or abstracts but do not query users. However, accurate clinical decision-making often requires additional patient-specific data obtained through differentiating questions. For example, clarifying whether a patient is a smoker or a nonsmoker or identifying ethnic background (Asian or White) may guide diagnostic pathways. If the system could formulate such discriminative questions and allow users to provide input—ideally guided by Bayesian theorem [18]—its clinical utility would be markedly enhanced. We did not observe this feature in the current implementation.

Future Directions

Once the pilot project is fully operational, the expert system can be refined along several pathways. First, the system should adapt its responses to the user’s professional role. For example, when handling a consultation request, it should provide only essential surgical information to an internist while delivering more comprehensive surgical details to a surgeon.

Second, although our study focused on airway diseases and the system demonstrated generally positive performance, its generalizability to other specialties remains uncertain. Disciplines such as oncology, where problems are more complex and controversies are common, may present challenges. Further research is required to determine whether the model can provide reliable and clinically meaningful output in these domains.

Finally, because decision-making, studying, and education are central components of our medical practice routines, integrating such a system into the existing health care information systems presents a logical next step. We expect that this integration could streamline clinical workflows and improve automation across multiple aspects of care delivery.

Conclusions

Our findings suggest that this approach is highly customizable, cost efficient, and accessible to clinicians and allied health care professionals without any computer coding experience in managing airway diseases. It provides convincing guideline-based recommendations, increases the staff’s medical literacy, and potentially saves physicians’ time spent on consultation. As AI tools gain widespread adoption, we believe that, if validated across additional medical specialties, this methodology could establish a new paradigm in clinical practice. While acknowledging the potential of AI, we emphasize the continued need for human judgment and oversight in evaluating and applying AI-generated recommendations.