This study was approved by the Seoul National University Bundang Hospital (SNUBH) Institutional Review Board (B-2011/646-104 for model development; B-2103-675-101 for physician comparison).

ICD-10 diagnosis code C56 was used to identify the study population. Data were retrospectively collected from patients admitted to the SNUBH for first-time ovarian cancer surgery between 2014 and 2021.

We divided the data into two parts by period: the internal data set (collected between January 2014 and September 2020) and the external validation data set (collected between October 2020 and February 2021) [24]. We chose the most recent 5 months of data as the external validation data set. The internal data set was used for training, validating, and testing the model, while the external data set was used for evaluating the final performance of the model and comparing the results with the physician’s assessment.

The exclusion criteria included readmission, admission with postoperative LOS <3 days, and patients who underwent surgery <20 times in the internal data set (Figure S1 in Multimedia Appendix 1). Postoperative LOS was chosen as the outcome variable because in-hospital LOS can be affected by several nonoperational factors such as patient characteristics or social circumstances, type of admission, patient place of residence, emergencies, or weekend admissions [9,25,26]. The postoperative LOS was defined as the number of days from the date of the index operation to the date of discharge, where the date of the index operation, denoted as day 0, was defined as the date in any of the operation-related nursing narratives. For example, if a patient was discharged on day 0, the postoperative LOS was 1.

We extracted nursing narratives chronologically. At SNUBH, nursing narratives are easily integrated into a structured database because individual features are mapped to unique codes [11,27]. For instance, the nursing narrative “checked the vital signs” is mapped to code N1, whereas “no dizziness” is mapped to code N2. A nurse enters patient statuses in the EHR system by searching nursing narratives using keywords such as “vital” or “dizzy” and selecting the appropriate nursing narratives from the list of related narratives. Some nursing narratives allow for the inclusion of additional information such as body temperature or free text [14]. Consequently, patient information was entered as a combination of unique codes (eg, N1, checked the vital signs, or N2, checked whether the patient felt dizzy), code entry time, and a specific value such as body temperature. These structured nursing narrative sets allowed us to retrieve patient information without the need for natural language preprocessing.

Prolonged LOS was defined as events ≥10 postoperative days, which was the third quantile of postoperative LOS in both the internal and external validation data sets. Our results showed that the volume of nursing narratives entered within 3 postoperative days was high and tended to decrease afterward; therefore, we decided to use patient information within 3 postoperative days, that is, from day 0 to day 2 (Figure S2 in Multimedia Appendix 1). The extracted time series of nursing narratives and corresponding unique codes were inverted to 3D arrays (patients, postoperative days, and nursing narratives’ unique codes).

The internal data set was randomly split, allocating 60% (n=192) of participants to the training set and 20% (n=64) of participants each to the validation and testing sets. The training set was used to train the models, the validation set was used to determine the values of the hyperparameters that increase the area under the receiver operating curve (AUC), and the test set was used to evaluate the performance of the best model. The best model was also applied to the external validation data set. Therefore, the performance of the best model was measured twice (the test set of the internal data set and the external validation data set). Furthermore, we compared the performance of the external validation data set with the physician’s assessment. The RETAIN model was constructed with two neural attentions that can identify influential nurse visits and meaningful features. The RETAIN model uses linear embedding to enhance interpretability. The contribution score was calculated using visit-level attention weights, variable-level attention weights, and embedding weights.

The default settings for RETAIN were used. L2 regularization for the final classifier weight, input embedding weight, and alpha-generating weight was set to 0.0001 for all models. Following a learning process using batch sizes of 8, 16, and 32, the model with the highest AUC performance on the test set was selected as the best model. If models had the same AUC value, the one with the highest sensitivity was selected as the best model.

The RETAIN model reported the contribution scores that represented the extent to which each feature contributed to the prediction. In this study, features with a high contribution score were associated with a high likelihood of prolonged LOS. We identified the input features with high contribution scores as influential features, which showed a significant difference between prolonged and short LOS via a t test with a P value cutoff of .05.

We compared the deep learning model and a physician’s assessment vis-à-vis their predictive capability for prolonged LOS. A gynecologic oncologist with 15 years of experience reviewed patients’ demographics, progress notes, surgical reports, and clinical notes available within 3 postoperative days. Blinded to the final discharge date, the physician predicted whether patients would experience prolonged LOS. The DeLong test was used to compare the AUCs of the deep learning model and a physician assessment [28].

Statistical analyses were performed using Python (version 3.9.13; Python Software Foundation), RETAIN (version 1.0; Edward Choi) [16], and R (version 4.0.5; R Foundation for Statistical Computer) software. The influential features were visualized using the ComplexHeatmap package in R software.

This study retrospectively enrolled 354 patients (n=320 in the internal data set and n=34 in the external validation data set; mean age 54, SD 13 years; Table S1 and Figure S1 in Multimedia Appendix 1). A total of 51,603 nursing narratives in the internal data set composed the model inputs. Patients in the prolonged LOS group were older (for instance, in the internal data set, the mean age was 57, SD 12 years and 52, SD 14 years in the prolonged LOS and short LOS groups, respectively; P=.002) and had higher total nursing narrative volumes (mean 188, SD 62 vs mean 150, SD 33 narratives within 3 postoperative days; P<.001). The. nursing narrative entries per nurse visit were similar (mean 5.6, SD 8.4 vs mean 5.9, SD 7.9 for the prolonged LOS and short LOS groups, respectively; P=.64), but more frequent nurse visits were observed in the prolonged LOS group (mean 33, SD 9 vs mean 25, SD 9 visits within 3 postoperative days; P=.03).

The experimental scheme is shown in Figures S2 and S3 in Multimedia Appendix 1. The RETAIN model was developed using the internal data set, while the model’s performance was calculated and compared to a physician’s expert clinical assessment using the external validation data set. The predictive contribution score derived from the RETAIN model indicates the probability of prolonged LOS. Patients with a final predictive score >0.5 were classified as expected prolonged LOS events. The RETAIN model reported the scores for each patient and nursing narrative; these scores were used to determine highly influential nursing narratives. We determined potent nursing narratives for each patient based on patient-wise normalization scores, after applying normalization using a patient-centric mean and SD. Thereafter, influential nursing narratives were defined as those consistently showing a statistically significant difference in the raw contribution score between the prolonged and short LOS groups.

Table 1 shows a comparison between the model’s performance and a physician’s expert clinical assessment, considering various nursing narrative sets. The model trained with nursing narratives showed an AUC of 0.81. The deep learning model performed better than the physician’s assessment (AUC 0.58; P=.02; Figure S4 in Multimedia Appendix 1).

Examples of contribution score graphs were visualized for patients in the prolonged and short LOS groups (Figure 1). As expected, the patients in the prolonged LOS group exhibited high contribution scores. Nurse visits on the first postoperative day (ie, day 1) were identified as highly influential because nursing narratives entered on that day exhibited higher contribution scores.

Highly influential narratives showing statistically significant differences in contribution scores between the prolonged and short LOS groups included the following: “confirmed by a doctor,” “injected intravenous patient-controlled analgesia [PCA],” “injected intravenous fluids,” “no PCA side effects,” “observed the pattern of Jackson-Pratt [J-P] tube drainage,” “patient’s pain in surgical area was tolerable,” “provided mental support,” “maintained J-P tube,” “maintained Foley catheter,” “no oozing in the drainage tube insertion area,” “measured body temperature,” “provided safety care,” and “notified a doctor” (Figures 2 and 3). The three most influential narratives (according to their lower P values) were “confirmed by a doctor,” “injected intravenous PCA,” and “injected intravenous fluids” (Table 2), whose contribution scores were visualized by t-distributed stochastic neighbor embedding (Figure 4).

Once the three most influential nursing narratives were identified, we further investigated the total number of entries and the first entry time since the handoff. The “confirmed by a doctor” narrative reoccurred in the prolonged LOS group (mean 5.8, SD 4.4 vs mean 3.1, SD 2.3 nursing narratives in the prolonged and short LOS groups, respectively) and was entered earlier (Table S2 in Multimedia Appendix 1). Conversely, the narratives “injected intravenous PCA” and “injected intravenous fluids” exhibited similar entry values but were entered a few hours later in the prolonged LOS group.

In this study, a RETAIN model was used to predict postoperative LOS using nursing narratives. The model achieved a higher AUC value of 0.81 compared to the physician assessment’s AUC of 0.58 (P=.02). Highly influential nursing narratives were identified that differed in their contribution scores between the prolonged and short LOS groups, including confirming by a doctor, administering intravenous PCA, and providing intravenous fluids.

To our knowledge, this is the first study to extract nursing narratives’ influential features by normalizing contribution scores estimated via RETAIN. By investigating these influential features, we discovered that the volume and timing of individual narratives are key factors. The likelihood of prolonged LOS increases if a physician must check the patient more often or if intravenous fluids or intravenous PCA are administered late. This degree of interpretability was not achievable in previous studies that relied on volume-centric statistical methods and conventional deep learning models.

This study demonstrated that nursing narratives can accurately predict the postoperative LOS of patients who underwent surgery for ovarian cancer. We implemented an interpretable deep learning model to identify highly influential nursing narratives. Notably, nursing narratives entered one day after surgery were the primary predictors for prolonged LOS.

Nursing narratives, serving as proxies for the care given to patients, demonstrated predictive value for LOS. Nursing narratives thus reflect the actions and interventions carried out by health care professionals. By identifying highly influential nursing narratives and presenting the different action timing and volume of each narrative, we enhanced the model’s interpretability and showed that the relevant nursing activities could serve as indicators for LOS.

These findings support other studies that have shown that nursing notes may predict short-term patient outcomes more accurately than physician notes [29,30]. Nurses frequently summarize patients’ situations by describing their symptoms, as well as their nursing actions and responses, without the restriction of structured forms [31-33]. Thus, nursing notes serve as a snapshot of patients’ current statuses and exhibit a higher degree of freedom compared to physicians’ notes, which provide a problem-focused summary. In a prospective cohort of patients who are critically ill, nurses predicted in-hospital mortality slightly more accurately than physicians, whereas the latter predicted long-term outcomes more accurately [29]. Huang et al [30] applied natural language processing to free-text nursing notes to predict multiple outcomes, including prolonged hospital stay or mortality, using the Multiparameter Intelligent Monitoring of Intensive Care III. This study also acknowledged the superior predictability value of nursing notes over physicians’ notes when using refined features within the first 48 hours of admission. However, none of these studies presented the additional interpretation of specific nursing notes.

Furthermore, this study showed that the total volume of nursing narratives is a significant factor for prolonged LOS, which was consistent with previous studies conducted in different settings. Schnock et al [34] have conducted a multicenter qualitative study in intensive and acute care units to discover nursing documentation patterns indicating recovery patterns. Woo et al [35] have used the natural language processing of nursing notes from patients admitted to home care and found that the frequency of wound infection–related text in nursing notes increased before hospitalization or emergency department visits. However, these studies faced a common barrier to using nursing notes: the extraction of standardized information. Accordingly, there is a significant need for health care providers to standardize nursing assessments and free-text notes [30].

We showed that the nursing narratives “confirmed by a doctor,” “injected intravenous PCA,” and “injected intravenous fluids” were relevant to a prolonged stay for patients with surgical procedures. These narratives suggested that a patient’s condition is complicated, and additional support for pain management or fluid management was required. Timely communication and collaboration between nursing and medical staff, effective pain management, and appropriate fluid management are important considerations in surgical patient care, which can impact the LOS and overall patient outcomes.

This study had several limitations. First, this study was based on data from a single-hospital EHR system. The EHR system at SNUBH allows for the standardization of nursing narratives, which enables the creation of a structured database. However, in most hospitals, free-text nursing notes are common; therefore, the preprocessing of natural language is required to generalize this study’s findings. As a starting point, it is worthwhile to examine the highly influential nursing narratives identified in our study. Second, we chose nursing narratives entered within a 3-day postoperative interval, which can be shortened in future studies. For example, 2-day postoperative data have been used in several studies [9,30]. Furthermore, a strategic patient care plan that combines a short-interval model and a long-interval model could be developed. Third, this study’s sample size was small, while the model was developed in a single-disease setting. To consider the dependency of nursing narratives according to different surgery and patient settings, transfer learning (in which a model trained in a larger population is fine-tuned with an independent surgery setting) can be considered. Future studies with multiple hospital settings and multimodal features are required [36]. Fourth, like other machine learning models, training and testing a model requires a large amount of data, and multiple validation sets are needed to avoid overfitting [37]. In addition, as the RETAIN model receives input values for each variable and visit level, it may be difficult to apply the model to unstructured data such as free text or data that cannot be classified by date. Finally, it is important to acknowledge that physician assessments were done retrospectively, potentially not capturing dynamic clinical situations.

Collecting a larger data set that includes a wider range of patients and additional predictors such as laboratory data or comorbidity information is essential. We firmly believe that integrating nursing narratives with broader information, including physician assessments, can lead to a better prediction model.

In this study, an interpretable prediction model for a longer postoperative LOS was developed using nursing narratives. The day after surgery was the most critical time for prediction, and influential nursing narratives were revealed. Although nursing narratives serve as proxies for the care given to patients, our study suggests that they have the potential to be predictors for LOS. The developed model can help identify patients with a prolonged hospital stay at the right time, thereby improving patient care and reducing hospital management burden. To strengthen the evidence supporting the predictive value of nursing narratives, either alone or in combination with broader information such as physician assessment, a larger data set would be beneficial.

Our study highlights that nursing narratives are predictors for prolonged LOS in patients undergoing ovarian cancer surgery. We emphasize the comprehensive nature of nursing actions and their timing in predicting patient outcomes and suggest methods to incorporate into a prediction model.