This study was approved by the institutional review committees of Severance Hospital (SH; 4-2021-1209), Gangnam Severance Hospital (GSH; 3-2021-0005), Konyang University Hospital (KYUH; 2021-10-003-001), Ajou University Hospital (AJUH; AJIRB-MED-MDB-21‐676), Seoul National University Cancer Hospital (SNUH; E-2207-151-1342), and the National Cancer Center (NCC; 2022-0184). All retrospective data were anonymized and appropriate measures were taken to protect participant information.

This retrospective observational cohort study analyzed the EHRs from 6 hospitals in South Korea between 1994 and 2021 to predict AKI. The EHRs were converted to OMOP-CDM (Observational Medical Outcomes Partnership Common Data Model) version 5.3.1. The 6 hospitals were SH, GSH, KYUH, AJUH, SNUH, and the NCC. An overall diagram of the cohort composition is provided in Figure 1.

In our cohorts, we adopted criteria based on serum creatinine (SCr) to define AKI according to the Kidney Disease: Improving Global Outcomes (KDIGO) Clinical Practice Guidelines and the previously defined AKI classification stages mapped in the “injury” category [24-26]. The criterion is an increase in the SCr level to 2 times the baseline value. As an alternative to the baseline SCr levels, we defined the upper limit of normal (ULN) value of SCr as 1.2 mg/dl [27].

The targeted drugs were selected from 5 medications associated with a high risk of AKI according to the US Food and Drug Administration (FDA) and previous studies: acetaminophen; vancomycin; 2 nonsteroidal anti-inflammatory drugs (NSAIDs), naproxen and celecoxib; and 1 antiviral drug, acyclovir [21,22].

The inclusion criteria were as follows: (1) the target drugs were administered, (2) the patient had a visit record of at least 30 days prior to the observation period, and (3) the patient underwent at least 2 SCr tests during the preobservation period (0‐60 days before the study). The exclusion criteria were as follows: (1) the patient had at least 1 SCr test outside the ULN value in the preobservation period. Participants were divided into case and control cohorts based on whether they met previously defined AKI criteria for 60 days after taking the first medicine. The observation period refers to the time range before and after the initial medication intake for each patient within the cohort. The cohort definitions were created using ATLAS, a web-based tool (Observational Health Data Sciences and Informatics), and are available as JSON files on GitHub [28].

To adjust for differences between cases and controls to reduce the effect of confounding variables, we used propensity score matching (PSM). Covariates included were age, sex, and SCr value at baseline. We normalized the covariates by applying standard scaling to ensure consistent dimensions across variables. A propensity score for each patient was generated by logistic regression. Patients were matched in a 1:3 ratio using a K-nearest neighbor (K-NN) algorithm using the Python scikit-learn library.

Candidate predictors were extracted from several key domains within the CDM containing per-patient observational data using structured query language tools in Python. Age and sex were used in the person domain, clinical laboratory tests in the measurement domain, medications in the drug exposure domain, diagnostic records in the condition occurrence domain, and surgical/procedure records in the procedure occurrence domain. Lab tests were treated as continuous variables, while other medications, conditions, and procedures were treated as binary variables. Statistical methods were used to select the variables. To identify predictors, we tested the statistical significance of the difference between the enrollment time of the cohort and the onset date of AKI using a 2-tailed paired t test and the McNemar test for continuous and dichotomous variables, respectively. To create a time series table, the candidate variables were pivoted into columns and dates were placed into rows. Missing values were handled in the following ways: forward fill for laboratory tests and diagnoses and zero fill for medications and treatments. The window size for predictions used 4-week sequence data and was processed by shifting the data of the prediction cycle by 2 weeks.

LSTM models based on recurrence have been designed to process time series data [29]. Attention-based LSTM models were initially proposed for learning words and the relationships between words in natural language processing [30,31] and later evolved into a key component of deep learning, becoming one of the methods used to provide interpretations, including importance scores for predicted outcomes.

As an advanced LSTM model, we used the IMV-LSTM module for the learning model, which is a multivariate LSTM neural network for the prediction and interpretation of multivariate time series [23]. This model improves on the LSTM-attention model, which can predict variable importance using multivariate inputs to configure variable-wise hidden states and mix both temporal and variable levels of attention for improved interpretability. The model was trained for 200 epochs with a batch size of 64 and a learning rate of 1e-3. An Adam optimizer was used with early stopping after 20 epochs. The data set was divided into training, test, and validation sets at a 6:2:2 ratio. Prediction performance was evaluated using the area under the receiver operating characteristic curve (AUROC) value. Additionally, we used the accuracy, precision, F₁-score, and area under the precision-recall curve (AUPRC) to ensure robustness for unbalanced data.

In this study, AKI prediction models were created for each hospital and drug. Each model had a different selection of candidate variables. To interpret the predictors in each model, variable- and temporal-wise attention scores were extracted from all trained models. These scores were then aggregated by calculating the overall temporal attention score, which was obtained by taking the weighted average of the temporal attention value over the attention value for each predictor variable. The resulting scores were plotted as heat maps for interpretation.

We used statistical packages based on Python and R (R Project for Statistical Computing) for the statistical analysis. First, to compare the AKI and non-AKI groups, we calculated significance using the χ² test for categorical variables and an independent-sample 2-tailed t test for continuous variables. Second, to identify differences in the pattern of AKI occurrence between cohorts and drugs, a histogram was plotted for patients in each cohort from the date of cohort entry (the first day of administration of the target drug) to the date of AKI occurrence. Differences between drugs were analyzed using an independent-sample2-tailed t test. Third, we compared the distribution of the aggregated temporal attention scores with the actual trained data with box plots of the data for 4 weeks at 1-week intervals. A repeated ANOVA test was performed to identify temporal differences.

The demographics of the 31,012 patients without AKI and the 8643 patients with AKI across the 6 hospitals after PSM are shown in Table 1.

By hospital, the cohort consisted of 14,046 patients from SH, 2180 from GSH, 2493 from KYUH, 4331 from AJUH, 13,174 from SNUH, and 4431 from NCC after matching the propensity scores. Propensity matching was performed using age, sex, and SCr levels at baseline. As for the changes in covariates, the difference in mean age decreased from 6.39 (60.65 – 54.26) to 0.44 (60.65 – 60.21) years, the difference in male ratio decreased from 16.49% (63.17% – 46.68%) to 7.02% (63.17% – 56.15%), and the difference in SCr at baseline decreased from 0.14 mg/dL (0.71 – 0.58 mg/dL) to 0.03 mg/dL (0.71 – 0.67 mg/dL) (Table S2 in Multimedia Appendix 1). There were still statistically significant differences in PSM age (60.66, SD 15.86 vs 60.22, SD 15.94 years; P=.03), gender (63.17% vs 56.15% male; P<.001), and SCr at baseline (0.71, SD 0.61 vs 0.68, SD 0.41 mg/dL; P<.001). Patients who developed AKI had more severe neoplasms (ie, active cancers; 70.6% vs 44.26%; P<.001) and chronic liver disease (15.83% vs 6.55%; P<.001). Moreover, the analysis revealed the following differences: sepsis (6.5% vs 1.94%; P<.001), diabetes mellitus (19.43% vs 14.57%; P<.001), hypertension (27.81% vs 23.5%; P<.001), anemia (9.81% vs 4.8%; P<.001), and heart failure (4.79% vs 3.07%; P<.001). There was no significant difference between hypotension (0.24% vs 0.21%; P=.56), potassium (4.11, SD 0.57 vs 4.11, SD 0.51 mmol/L; P=.48), or renal artery stenosis (0.07% vs 0.03%; P=.13). Hypoalbuminemia, obesity, peripheral vascular disease, renal artery stenosis, liver dysfunction, and prior kidney surgery had low incidence rates (<2%).

To analyze the differences in drug patterns at the time of AKI occurrence, we assessed the pattern for each drug. The median number of days for the occurrence of AKI among the drug and cohort patients in the entire hospital was 17 (IQR 7-33 days). Vancomycin appeared after a median period of 12 days, followed by naproxen (18 days), acetaminophen (19 days), celecoxib (22 days), and acyclovir (23 days). When comparing the IQR values, celecoxib (10‐41 days) and Acyclovir (10‐41 days) showed a relatively broad distribution, whereas acetaminophen (9‐34 days) and naproxen (8‐34 days) were distributed over 25 days. Vancomycin (5‐25 days) exhibited the narrowest distribution. Celecoxib and acyclovir tended to be relatively distributed compared to acetaminophen, vancomycin, and naproxen. We compared the onset times of all drug pairs and hospital pairs to check the similarity in AKI occurrence (Figure 2). The patterns between specific drugs was similar for celecoxib and acyclovir (P=.88) and for acetaminophen and naproxen (P=.57). The patterns between hospitals were similar for SH and AJUH (P=.98), SH and GSH (P=.36), GSH and AJUH (P=.42), GSH and NCC (P=.24), and SNUH and NCC (P=.26).

The AUROC for each drug and hospital to evaluate the AKI predictive model, based on respective test sets (internal validation) is shown in Figure 3. A total of 26 trained models achieved a high AUROC value, of 0.92 on average, with each verification data set. In addition, among the averages of the drugs, acyclovir had the highest average AUROC score of 0.94, followed by acetaminophen (0.93), vancomycin (0.92), naproxen (0.90), and celecoxib (0.89). The highest AUROC value (0.97) was observed for the model of SH’s celecoxib and acyclovir, SNUH’s vancomycin, and KYUH’s acyclovir prescription patients.

Multimedia Appendix 2 presents data on the precision, accuracy, F₁-score, and AUPRC of each predictive model. Overall, the average accuracy of the AKI prediction models was 0.88, whereas the average AUPRC and F₁-scores were both 0.78. The acyclovir prescription model achieved the highest accuracy score (0.91), followed by vancomycin (0.90), acetaminophen (0.89), naproxen (0.89), and celecoxib (0.86). Individually, the acyclovir SH model showed the best performance, with an AUPRC of 0.92 and an accuracy of 0.91.

To interpret the AKI prediction model, we demonstrated the temporal attention values of each contributing variable in the 4 weeks prior to AKI onset, which were weighted aggregates from the model for each drug and hospital, as shown in Figure 4A. The temporal change pattern of the actual data corresponding to each variable in the 4 weeks prior to AKI onset is shown in Figure 4B. We also confirmed the difference in the distribution of highly important features between the case and control data using a 1-way ANOVA. The attention scores for all variables across all hospitals are detailed in Multimedia Appendix 3

The last week of lymphocytes (attention score at −1 week: 0.41) and the second week of calcium (attention score at −3 weeks: 0.41) showed the highest attention scores, followed by albumin (attention score at −1 week: 0.37; attention score at −4 weeks: 0.37), hemoglobin (attention score at −4 weeks: 0.37), and cholesterol (attention score at −4 weeks: 0.37). In Figure 5, the distribution of data by variable between the 2 groups was confirmed using actual data. There was a difference in data distribution between the case and control groups from the beginning. The values of lymphocytes, albumin, and hemoglobin in the case group decreased over time, while urine pH and prothrombin time in the case group tended to increase over time.

In this study, we developed time series–based IMV-LSTM models to predict AKI in patients taking specific nephrotoxic drugs using CDM-based DRNs in a 6-hospital EHR-based system. The principal findings are as follows: first, this study provides an interpretation of the temporal importance of variables for predicting AKI, and the models also achieved high performance, with an average AUC of 0.92%. Second, our study is a scalable multicenter study using a DRN, which can contribute to understanding drug-induced AKI. To the best of our knowledge, this is the first study to build an AKI prediction model by applying a time series–based IMV-LSTM model to a CDM using EHR data from 6 hospitals.

We established a retrospective cohort of patients who took nephrotoxicity-inducing drugs at 6 hospitals. With respect to demographic characteristics, we observed variations in the overall patient count and prevalence of comorbidities when comparing individuals with AKI and without AKI across different hospitals. Nevertheless, the majority of patients who developed AKI at most hospitals were older than 60 years and had a high prevalence of comorbidities, including cancer (n=6001, 69.43%), hypertension (n=2404, 27.81%), diabetes (n=1679, 19.43%), and chronic liver disease (n=1231, 14.24%), which is consistent with findings reported in previous studies [32-35].

The pattern of each drug’s association with AKI (Figure 5) showed that the median number of days for AKI onset when using nephrotoxic drugs was 17 (IQR 7-33) days. The onset occurred earliest with vancomycin (12, IQR 5-25 days) and latest with acyclovir (23, IQR 10-41 days). In previous studies [36,37], the time to onset of vancomycin-induced AKI showed a similar pattern to our results. We also found differences in the AKI onset between different classes within the same NSAID, and the multicenter AKI cohort showed similarities between hospitals. The finding of similar patterns in the AKI onset in the multicenter cohort supports the reliability of the AKI cohort and increases the explanatory power of AKI prediction models.

AKI is common among inpatients [38,39]. Previous models predicted AKI in the intensive care unit (ICU) and in surgical patients during hospital admission. For example, Zimmerman et al [40] predicted the occurrence of AKI in ICU inpatients (AUC 0.783), and Tseng et al [41] developed a predictive score for the development of AKI after cardiac surgery (AUC 0.839). Hsu et al [42] developed a risk score function for community-acquired AKI for inpatients (AUC 0.818). Koyner et al [10] developed a model to predict AKI in hospitalized patients (AUC 0.90). Despite this progress, few studies have applied time series deep learning to provide a temporal interpretation of drug-induced AKI, and our model stands out because it can predict nephrotoxic drug–induced AKI in a diverse hospital population.

This study achieved improved performance compared to previous AKI studies using recurrent neural networks (RNNs) [21,43-45]. Our model improves performance up to an AUC of 0.97 and an overall average of 0.92, which outperforms previous studies showing AKI prediction with RNN-based methods by Kim et al [43] in hospitalized patients (AUC 0.927), Rank et al [45] in cardiac surgery patients (AUC 0.893), and Xu et al [44] in inpatients (AUC 0.908). These results show promise for our model as a tool to predict AKI and facilitate early intervention and mitigation strategies for patients.

This study also provides additional interpretations regarding the temporal importance of features for AKI prediction. Some studies have reported information on interpretability or provided information about the interpretability of variables at the feature importance level [46]. However, our results show the importance of variables and the temporal importance of variables in the development of AKI. In this study, we highlight the vital role of temporal patterns of various indicators, such as lymphocytes, calcium, albumin, hemoglobin, and cholesterol, in predicting disease states, particularly the onset of AKI. The temporal pattern of lymphocytes increased gradually, peaking 1 week before AKI onset. The use of lymphocyte and neutrophil counts as predictive factors for AKI is consistent with other studies [47,48]. Calcium shows a pattern of peaking 3 weeks before AKI onset, and Prior studies showed an association between impacted calcium metabolism and AKI [49,50]. Albumin shows the highest pattern 1 and 4 weeks before onset, and low serum albumin levels (hypoalbuminemia) are a predictor of AKI [48,51,52]. Hemoglobin shows a pattern with a peak 4 weeks before onset, and previous studies have shown that the risk of AKI increases stepwise with a further decrease in hemoglobin concentration [53]. Temporal variations in variables based on reported laboratory data for the early detection of AKI emphasize the importance of monitoring and early intervention in populations.

In addition, this retrospective study can be followed by a subsequent study to validate the practicality of the AKI prediction model in clinical practice by applying it to a prediction system in a hospital EHR.

This study has several limitations. First, because we used the CDM, it does not reflect the full range of clinical data. For example, we could not include admission records, which would have revealed a patient’s condition. However, the use of CDM data allowed for a multicenter study that could be easily extended to other institutions that have converted to the CDM. Second, this was a retrospective study and could not address the underlying causes of AKI. Therefore, prospective studies are needed for validation with actual clinical data. Third, as with all retrospective studies, there may be unintentional patient selection bias and unaccounted-for confounders. However, to compensate for these limitations, we tried to equalize the distribution of patient characteristics through PSM. We also used a limited follow-up period to minimize the impact of these factors.

This study demonstrates the high performance of the IMV-LSTM method for AKI prediction using hospital EHR-based time series data. Our model can provide real-time assessment of AKI occurrence and individualized risk factors for AKI using time series data. We also demonstrated the robustness of our model through multicenter validation using a CDM through a DRN of 6 hospitals in South Korea, which also proves that scalability to other institutions that are converted to the CDM is possible. This may provide an objective quantitative tool for identifying patients at risk of developing AKI.