Our article follows the guidelines of the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis (TRIPOD) consensus document. This study is a retrospective electronic health record (EHR) data analysis. This retrospective observational study used EHR data collected at a tertiary acute care hospital with over 2500 medical conditions in Seoul, Republic of Korea. To develop a readmission early prediction model for readmission of high-risk discharge patients, we employed a retrospective study design utilizing nursing data.

We extracted medical record data of 12,977 patients at S Hospital in Seoul, Korea, for 25 months from January 2018 to January 2020. As a study to confirm the EHR data from the time of admission to the time of discharge of the patient, the information collected until January 31, 2021, was used to track readmission within 30 days. Data were requested from S University Hospital via the SOBIG data portal using subject inclusion criteria. The big data team extracted and received data from the first hospitalization of adult patients hospitalized for 6 major disease groups (chronic obstructive pulmonary disease, liver disease, diabetes mellitus, lung disease, heart failure, and chronic kidney disease) for 2 years. Hospital S is a high-level general hospital with about 2500 beds, with approximately 2000 to 3000 surgeries per month and more than 10,000 outpatient visits per day. We secured medical records of medical institutions where many patients are frequently hospitalized and readmitted and used them to develop a prediction model.

We used various types of EHR data from the participants, including during hospitalization and follow-up data within 30 days of discharge point. Analysis in this study included general, clinical, and nursing data recorded after the direct assessment of patients.

The data inclusion criteria are as follows: patients were older than 18 years at admission. The disease group selection of the subjects was based on the most recent 2021 HIRA (Health Insurance Review and Assessment Service) data, and the discharged patients from the top 6 main diagnosis groups with high readmission rates were selected. One patient had only one record.

The data exclusion criteria are as follows: patients who died during hospitalization. Data resulting in death were excluded from discharge records.

The ethics review committee of Severance Hospital approved this retrospective data analysis study (DRB approval number 4-2021-1334). Severance Medical Center’s guidelines for using EHRs require the data to be provided in Excel format with patient identification information removed. Data analysis was performed based on this data.

EHR data were analyzed to predict patient readmissions. Readmissions, the primary outcome of interest in the study, were assessed as unplanned readmissions within 30 days [13]. The study aimed to determine a patient’s likelihood of readmission to develop an early discharge care plan. To determine the likelihood of patient readmission, we classified the target variable Y as yes (1) or no (0). Predictor variables (X) necessary to confirm the presence or absence of rehospitalization, which is the target variable of the patient, including sex, age, admission days, diagnosis group, number of ICD-10 codes, admission and discharge medications, state of consciousness, BMI, falls and pressure ulcer risk, vital signs, literacy, economic status, functional and emotional evaluation, number of nursing diagnosis records, presence of the caregiver, admission via the emergency room, health behavior habits (smoking, drinking, regular exercise, sleep disorder, and nutritional status), blood test values (levels of sodium, hemoglobin, potassium, aspartate aminotransferase, glucose, and creatinine; and white blood cell counts) were included in the analysis. Utilizing coefficient of variation (CV) values, Model 2 incorporated patient data from admission to discharge to capture data variability.

Among the various variables, the following are included as nursing data: falls and pressure ulcer risks, vital signs, literacy, economic status, functional and emotional evaluation, number of nursing diagnosis records, presence of the caregiver, health behavior habits (eg, smoking, drinking, regular exercise, sleep disorder, and nutritional status), and ward severity. The ward severity score is a score given by the nurse in charge of the ward to each patient by identifying the number of interventions in the areas of hygiene, nutrition, elimination, exercise and activity, education and counselling, emotional support, measurement and observation, communication and alertness, treatment and testing, medication, interdepartmental coordination of patient care, discharge, and power management. The modified Barthel index scale is an ordinal scale that measures an individual’s ability to complete activities of daily living. It consists of 10 items and has a total score of 100. However, the organization that collected the data evaluated 11 items by adding an item on the ability to use walking aids other than wheelchairs, and the total score was 105. All of these nursing data variables include information recorded by the nurse during the patient’s admission to the hospital to indicate the patient’s condition. We built a model to predict unexpected readmissions, divided into Model 1 and Model 2. Model 1 entered possible variables based on data from the first day of hospitalization, and Model 2 selected variables based on data from the entire period of hospitalization. All variables were selected based on prior literature and entered through stepwise feature selection [14,15].

Table 1 displays all the input variables according to Models 1 and 2.

Machine learning is a field of artificial intelligence that studies the development of computer algorithms and techniques that automatically improve through experience [16]. Machine learning is divided into supervised learning and unsupervised learning and predicts outcomes through supervised learning. There are various types of machine learning. A representative example is a decision tree, a graph representing a selection and its result in the form of a tree. The nodes of the graph represent events or selections, and the edges of the graph represent decision rules or conditions. Each tree consists of nodes and branches. Each node represents an attribute of a group to be classified, and each branch represents a value that the node can take. It is an algorithm suitable for a rehospitalization prediction model that mainly predicts the presence or absence of rehospitalization. The boosting algorithm is a family of algorithms that transform weak learners into strong learners. Boosting is an ensemble learning technique used to reduce bias and variance. A weak learner is defined as a classifier, and a strong learner is a classifier that is randomly correlated with the actual classification. Many algorithms with high accuracy and predictive power are being developed in prediction algorithms. Bagging is applied where the accuracy and stability of machine learning algorithms must be increased, and it can be applied to classification and regression and helps reduce variance and handle overfitting. The random forest algorithm is an algorithm that creates a decision tree using this bagging method. It is an algorithm that can solve the overfitting problem that a single decision tree can have by creating various subdatasets through bootstrapping and having multiple decision trees learn each dataset and collate the results [6]. In this study, we used a supervised learning method through 6 algorithms that are considered most suitable for predicting readmission among machine learning algorithms: logistic regression, decision tree, random forest, CatBoost, XGBoost (extreme gradient boosting), and multilayer perceptron, and developed a readmission prediction model for high-risk patients. The dataset was split into 60% for training, 20% for validation, and 20% for testing, with all models trained 5 times and model performance evaluated using layered 5-fold cross-validation. We performed a grid search to find the most suitable hyperparameters for the model while training and evaluating the model, and then performed model fine tuning.

According to the data inclusion and exclusion criteria and missing value processing, data from 9028 patients were available for analysis. We constructed unexpected readmission prediction models by dividing them into Model 1 and Model 2. Model 1 implemented an early readmission prediction model based on data from the first day of hospitalization to predict readmission early. Model 2 implemented a model to supplement data that should have been included in Model 1 based on all the data. All machine learning–based predictive model development processes were conducted with the participation of artificial intelligence experts, who conducted the procedures together, provided advice, and reviewed them. The analysis in this study was performed using Python Language Reference, version 3.11.3. (Python Software Foundation).

After implementing 6 models, we selected the final model and evaluated each model’s confusion matrix, accuracy, precision, recall, F1-score, and area under the receiver operating characteristic (AUROC) curve.

We compared techniques using performance measures of different models. Confusion matrices were used to determine accuracy, recall, precision, and F1-score. True positive, true negative, false positive, and false negative were used to determine the performance of each model. True positive refers to cases where the model predicted yes, and the patient was actually readmitted. In contrast, true negative refers to cases where the model predicted no, and the patient was not readmitted. Conversely, false positive refers to cases where the model predicted yes, but the patient was not actually readmitted, and false negative refers to cases where the model predicted no, but the patient was not actually readmitted.

In predicting unplanned readmission within 30 days using data from 9028 high-risk discharged patients, machine learning methods (random forest and CatBoost models) showed better predictive power than traditional logistic regression methods, similar to other previous studies [17,18]. While previous studies have focused on developing a prediction model for a specific condition for a specific group of patients with a specific diagnosis [19], this study is significant in that it is a prediction model development study that focuses on the generalization process of predicting unplanned readmissions within 30 days by incorporating the top 6 diagnosis groups of high-risk discharged patients suggested by the Health Insurance Review and Assessment Service to predict the potential of readmission.

Many interventions aimed at reducing patients’ unplanned readmissions are often initiated just before or after inpatient discharge, which has little impact on improving the inpatient quality of care [20]. Providing early interventions related to readmissions during hospitalization with early discharge planning can reduce readmissions [21]. However, most readmission prediction models are built on posthospitalization data, often including information about the patient’s physical and diagnostic tests. Because clinically essential variables such as the length of stay, diagnosis codes, laboratory test results, and medications are usually entered just before discharge, readmission prediction models are primarily built on patient data immediately before or after discharge. The most widely used measures of readmission risk in US health care settings are the HOSPITAL score [22] and the LACE index [23], but both have the limitation that they are only available at the end of a hospital stay. When creating a prediction model that includes all variables, its prediction performance is good. However, it needs to be applied for more time to actual patients, making it challenging to support effective clinical decision-making. In this study, to overcome these limitations, we implemented two patient readmission prediction models, Model 1 and Model 2, to predict the readmission rate from the time of hospitalization and plan discharge care. Model 1, implemented based on the patient’s initial hospitalization data, and Model 2, implemented by adding variables that can complement Model 1, can help clinical decision-making by predicting the patient’s readmission rate earlier. Model 1, which utilized data from the first day of hospitalization, had an AUROC curve of 0.62 in our study. Although this performance is not exceptionally high, it is consistent with most previous studies that developed readmission prediction models using initial hospitalization data, which reported results of less than an AUROC curve of 0.75 [5,17,24,25]. Importantly, it enables the proactive identification of high-risk patients and the provision of timely and preventive therapeutic interventions early in their hospital stay.

Unplanned readmissions cause a heavy burden on individuals and society through unnecessary health care costs and resource use, and readmission management has been recognized as an essential issue in improving patient quality of care [26]. To reduce the cost of unnecessary readmissions and improve the quality of care for patients, the Korea Health Insurance Review and Assessment Service conducts a national risk-based readmission cost appropriateness assessment, and the US government has implemented the Hospital Readmission Reduction Program (HRRP) since 2012 to encourage the reduction of hospital readmissions by involving patients and caregivers in discharge planning [27]. However, limited health care resources make it challenging to provide discharge interventions to all patients, and screening high-risk patients for early discharge planning is essential. Evidence suggests that focusing interventions on high-risk patients can reduce the risk of hospital readmission within 30 days by 11‐28% [28-30], and it is essential to utilize nursing data that include holistic information from the patient’s initial hospitalization to develop a preventive care plan for high-risk patients. A previous study found the potential importance of utilizing nursing variables to identify risk factors for readmission [31]. The BMI variable at the top of the feature importance of Model 1 is the same as the result that it was highly related to the patient’s obesity level, cardiovascular problems, and readmission, similar to the previous study [32]. This study is significant in that it proved that BMI is the most simple and important screening variable related to the patient’s readmission in the early stage of hospitalization. In the future, it is necessary to recognize the importance of BMI screening and collect information related to the probability of early patient readmission [32].

Information collected directly about patients’ physical, mental, and social aspects by nurses, who interact most closely with patients in the hospital and provide care, is essential in predictive models for the early identification of high-risk discharges. In this study, we developed an early prediction model for unplanned readmissions using nursing data from patients in the early stage of hospitalization. In particular, we found that many nursing data variables were included in the high rankings among the important predictors in Model 1 developed using early hospitalization data. The high representation of nursing variables among the most influential predictors underscores the potential value of integrating nursing-specific information into risk assessment tools for hospital readmission. Further investigation into the specific contributions of these nursing variables may provide valuable insights for enhancing the accuracy and clinical utility of early readmission prediction models. Although most predictions of high-risk discharged patients are based on data at discharge, identifying high-risk patients should ideally be performed early enough to allow for therapeutic interventions during hospitalization, which was attempted in this study.

This study has several limitations. First, our prediction model, designed for adults aged over 18 years, does not apply to pediatric patients aged less than 18 years. Second, the model’s development was based on data from a single hospital in a specific region. Consequently, its applicability to readmission scenarios in different health care settings or countries may be limited. Furthermore, tracking patient readmissions poses a challenge, especially in cases where patients admitted to our hospital could have subsequent readmissions to different facilities. This potential for cross-hospital readmissions might not be fully captured in our model.

Additionally, as this was a retrospective study utilizing deidentified data, it limited our ability to validate the relationships between our findings and factors identified in other studies. Finally, there were limitations in implementing perfectly standardized data, as the data were manually entered as part of the medical and nursing data.

Our readmission prediction model can be used to predict and continuously monitor a patient’s risk of readmission during the entire hospital stay. It can be used as an early screening tool to assess the risk associated with a patient’s readmission.

This study developed two prediction models, Model 1 using initial hospitalization data and Model 2 reflecting the variability of variables from admission to discharge, for the early prediction of unplanned readmission within 30 days in high-risk discharged patients. Model 1 showed the best performance of the random forest model, with an AUROC curve of 0.95 for the training data and 0.62 for the test data. For Model 2, the CatBoost model performed the best, with an AUROC curve of 0.92 for the training data and an AUROC curve of 0.64 for the test data. The Model 1 and Model 2 predictive models allow us to predict the likelihood of readmission from the beginning of a patient’s hospitalization and proactively plan for early discharge. In addition, this study identified various significant predictors of unplanned readmission and identified changes in patients over time. The predictive model was developed and validated using data from 9028 discharged patients and can be applied to adult patients who represent the high-risk discharge population in tertiary care hospitals. However, further research and experimentation are necessary to implement the developed predictive models in clinical practice within an EHR environment for predicting early readmission and monitoring readmission risk. In particular, additional studies must be conducted to address the limitations of this monocentric study, consider the characteristics of medical data with numerous missing values, and validate the data sources for each variable.

Based on this study, future development of a systematic clinical decision support system that utilizes machine learning–based readmission prediction models with nursing data could enable real-time early risk prediction of patient readmission. This approach would facilitate the provision of preventive discharge services for high-risk patients and improve the quality of health care services.