Taichung Veterans General Hospital (TCVGH) is a 1530-bed medical center in central Taiwan with 7 ICUs comprising a total of 138 beds. We obtained access to the critical care database (AI-111010) of the AI Center of TCVGH. The following data were collected: basic information, disease severity, ventilator use, blood biochemistry, vital signs, catheter types, and medication records.

The study consisted of five major steps: (1) data collection in the ICU, (2) data preprocessing (data imputation and data sampling), (3) ensemble learning model construction, (4) final evaluation, and (5) implementation of explainable AI (Figure 1).

Based on clinical care experience, this study proposed an ensemble learning model that integrates two submodels—sedation and agitation—to classify events into oversedation (RASS -5 to -2), maintain range (clinically expected maintenance RASS -1 to 1), and agitation (RASS 2 to 4).

First, the sedation and agitation models were constructed using 4 ML algorithms: logistic regression, random forest, XGBoost, and LightGBM. The algorithm with the best performance was selected as the foundation for both the sedation and agitation models. Subsequently, the two submodels were combined into 2 sequential ensemble learning models: the sedation-first ensemble model and the agitation-first ensemble model. In the sedation-first model, the sedation model was first used to distinguish “oversedation” from “other,” and the remaining categories were then input into the agitation model to further differentiate between “maintain range” and “agitation.” Conversely, in the agitation-first model, the agitation model was first applied to separate “agitation” from “other,” and the remaining categories were then passed into the sedation model to classify “oversedation” and “maintain range.” (Figure 2).

We used confusion matrices and the receiver operating characteristic (ROC) curves as indicators to evaluate model accuracy, precision, recall, F1-score, and area under the curve. The ROC curves helped to compare sensitivity with specificity. Effective models exhibited high sensitivity and specificity, resulting in high area under the curve values.

Explainable Artificial Intelligence (XAI) was applied, making the ML system transparent. The top 20 features were selected, and Shapley additive explanations and partial dependence plots were used to visualize their contributions, aiding in understanding the model’s decision-making process. Clinical personnel could use this information to offer patient-specific evaluations or decision-making suggestions.

This study used statistical methods, including mean (SD) and t-test, for numerical data analysis and observation. These methods were used to describe dataset central tendencies, assess variability, and compare group differences. Proportions of each category were also calculated for categorical data. They helped to extract meaningful information and interpret research results.

This study was approved by the TCVGH Institutional Review Board (CE22484A). Informed consent was obtained, with participants given the option to withdraw at any time. For secondary analyses, original consent covered data reuse without additional approval. Data were anonymized to protect privacy, and strict security measures were applied.

This study collected data from adult patients (aged≥20 years) admitted to the ICU at TCVGH between January 1 and December 31, 2020, with an ICU stay lasting more than 24 hours. Every 4-hour RASS assessment (with increased frequency depending on the patient’s condition) was considered as a classification event, with events marked as not assessable excluded due to concurrent procedures. Since the average ICU stay did not exceed 30 days, events from ICU stays longer than 30 days were also excluded. A total of 121,306 events were collected, with an average of 108 events per patient (range: 6 to 186). The machine learning (ML) model was trained using data from the 8 hours prior to each event, including ventilator parameters, vital signs, and medication records, along with laboratory biochemical data from the previous week (Figure 3).

Statistical analysis indicated that nearly all features significantly affect the level of agitation-sedation. Patients with oversedation typically exhibited higher disease severity, lower blood oxygen levels, and a higher proportion of invasive ventilator use. In contrast, patients with agitation showed higher vital sign values and a greater proportion of sedative medication usage (Table 2).

The top 4 features that contribute the most to sedation classification are the use of an invasive ventilator (invasive ventilation), APACHE II score, lactate level (LACTATE), and average airway pressure (PAW). A high APACHE II score indicates a high likelihood of oversedation (Figure 4A). The dependency graph of the first 4 features represents how each feature affects the classification results. In most cases, an APACHE II score>29 was positively associated with oversedation (Figure 4B).

Patients who used sedatives were more likely to experience agitation. Patients with a high hemoglobin level were more likely to experience agitation. The dependence plots of the top 4 features indicated that sedative use was positively correlated with agitation. Hemoglobin levels >10 g/dL and≤10 g/dL were positively correlated with agitation and maintained sedation range, respectively (Figure 5).

Overall, patients on mechanical ventilation were mostly sedated, with those in the maintain range exhibiting more stable blood test results and vital signs compared to oversedated or agitated patients.

This study successfully developed a model for automating RASS-like agitation-sedation evaluation. Automated agitation-sedation evaluation could be an alternative to RASS and play a crucial role in enhancing ICU efficiency, ultimately improving health care outcomes, care quality, and patient safety.

Machine learning aids ICU personnel in the early detection of high-risk events [14]. Previous studies have used ML to predict mortality rates in ICU patients with acute kidney injury, predict postoperative sepsis mortality rates, and forecast extubation failure in the ICU [15-17]. However, studies on ML for agitation-sedation evaluation in ICU patients are limited.

Zhang et al [18] ensemble 4 machine-learning methods for predicting agitation in ventilated patients under light sedation for 24 hours. However, their ensemble model was limited to predicting agitation in patients with invasive ventilator support under light sedation for 24 hours. Other researchers have proposed using patient body and facial image monitoring for agitation detection [19,20]. However, image monitoring faces challenges such as data acquisition, clinical environment influences, workflow integration, and system installation. Therefore, in addition to the imaging model developed by our research team [21], we have created another model using commonly available feature data in most hospitals, serving as an alternative solution when image monitoring is not feasible.

The study employs 2 ML models for ensemble learning to automate RASS assessment. By using undersampling and adjusting the classification sequence, the sensitivity for detecting agitation is enhanced. Although models with higher sensitivity may reduce the accuracy of classifying over-sedated patients, they perform better in mitigating immediate risks. Traditional methods, which involve manual assessments every 4 hours, result in insufficient monitoring, especially for patients who may experience multiple episodes of agitation within a short period. In contrast, our health care information system can transmit data in real time, perform inference every hour, and adjust inference frequency based on clinical needs, enabling more intensive monitoring. This allows clinicians to track patient conditions and respond promptly to changes continuously.

Understanding algorithmic predictions is crucial in clinical practice. Due to the lack of explanations in the decision-making process, clinicians often distrust black-box models. Explainable artificial intelligence enhances transparency, aiding in the development of reliable decision models [22-24]. Continuous analgesics and sedatives ensure optimal gas exchange between patients and ventilators, making deep sedation important during this period [25]. For patients with stable parameters such as hemoglobin, hematocrit, blood oxygen saturation, and creatinine, the goal is maintaining light sedation (RASS score 0 to −2) [26]. It has been demonstrated that the interpretability of the model aligns with clinical experience.

This study acknowledges its limitations, particularly the imbalance in case numbers due to the high risk of agitation in patients. Especially for agitated individuals, the model’s precision of 0.03 suggests there is room for improvement. In future clinical applications, we plan to adjust the decision threshold based on specific needs to balance sensitivity and specificity and reduce false positives. Additionally, the study lacks observations of potential drug overdoses at different sedation levels. Future efforts will focus on integrating this model with digital imaging monitoring and a comprehensive drug dosage system. Real-time monitoring will help identify patient conditions, guide prescription adjustments, and accelerate recovery, ultimately supporting early intervention, ensuring patient safety, and improving the quality of intensive care.

This study proposes using ML technology to achieve automated RASS-based assessments in ICU settings, enhancing clinical efficiency, and patient safety. Our integrated learning model, combined with the hospital information system, enables real-time data transmission, supports intensive monitoring, and facilitates continuous tracking of patient conditions. The system automatically categorizes patients into three groups, significantly improving sensitivity in detecting agitation categories. This innovative approach not only alleviates the workload of health care professionals but also advances the precision and intelligence of critical care management.