Published on in Vol 12 (2024)

Preprints (earlier versions) of this paper are available at, first published .
Time Series AI Model for Acute Kidney Injury Detection Based on a Multicenter Distributed Research Network: Development and Verification Study

Time Series AI Model for Acute Kidney Injury Detection Based on a Multicenter Distributed Research Network: Development and Verification Study

Time Series AI Model for Acute Kidney Injury Detection Based on a Multicenter Distributed Research Network: Development and Verification Study

1Department of Biomedical System Informatics, Yonsei University College of Medicine, , Seoul, , Republic of Korea

2Division of Allergy and Immunology, Department of Internal Medicine, Yonsei University College of Medicine, , Seoul, , Republic of Korea

3Institute of Allergy, Yonsei University College of Medicine, , Seoul, , Republic of Korea

4Medical Informatics Collaborative Unit, Department of Research Affairs, Yonsei University College of Medicine, , Seoul, , Republic of Korea

5Department of Computer Engineering, Gachon University, , Seongnam, , Republic of Korea

6Transdisciplinary Department of Medicine & Advanced Technology, Seoul National University Hospital, , Seoul, , Republic of Korea

7Healthcare AI Team, National Cancer Center, , Goyang, , Republic of Korea

8Department of Biomedical Informatics, Ajou University School of Medicine, , Suwon, , Republic of Korea

9Healthcare Data Science Center, Konyang University Hospital, , Daejeon, , Republic of Korea

10Department of Biomedical Sciences, Ajou University Graduate School of Medicine, , Suwon, , Republic of Korea

11Department of Statistics, Korea University, , Seoul, , Republic of Korea

*these authors contributed equally

Corresponding Author:

Yu Rang Park, PhD

Background: Acute kidney injury (AKI) is a marker of clinical deterioration and renal toxicity. While there are many studies offering prediction models for the early detection of AKI, those predicting AKI occurrence using distributed research network (DRN)–based time series data are rare.

Objective: In this study, we aimed to detect the early occurrence of AKI by applying an interpretable long short-term memory (LSTM)–based model to hospital electronic health record (EHR)–based time series data in patients who took nephrotoxic drugs using a DRN.

Methods: We conducted a multi-institutional retrospective cohort study of data from 6 hospitals using a DRN. For each institution, a patient-based data set was constructed using 5 drugs for AKI, and an interpretable multivariable LSTM (IMV-LSTM) model was used for training. This study used propensity score matching to mitigate differences in demographics and clinical characteristics. Additionally, the temporal attention values of the AKI prediction model’s contribution variables were demonstrated for each institution and drug, with differences in highly important feature distributions between the case and control data confirmed using 1-way ANOVA.

Results: This study analyzed 8643 and 31,012 patients with and without AKI, respectively, across 6 hospitals. When analyzing the distribution of AKI onset, vancomycin showed an earlier onset (median 12, IQR 5-25 days), and acyclovir was the slowest compared to the other drugs (median 23, IQR 10-41 days). Our temporal deep learning model for AKI prediction performed well for most drugs. Acyclovir had the highest average area under the receiver operating characteristic curve score per drug (0.94), followed by acetaminophen (0.93), vancomycin (0.92), naproxen (0.90), and celecoxib (0.89). Based on the temporal attention values of the variables in the AKI prediction model, verified lymphocytes and calcvancomycin ium had the highest attention, whereas lymphocytes, albumin, and hemoglobin tended to decrease over time, and urine pH and prothrombin time tended to increase.

Conclusions: Early surveillance of AKI outbreaks can be achieved by applying an IMV-LSTM based on time series data through an EHR-based DRN. This approach can help identify risk factors and enable early detection of adverse drug reactions when prescribing drugs that cause renal toxicity before AKI occurs.

JMIR Med Inform 2024;12:e47693



Acute kidney injury (AKI) is associated with a mortality rate of 40%‐70% in hospitalized patients who develop AKI and causes significant kidney damage even after recovery, leading to dialysis, longer hospital stays, and increased costs of care [1-4]. Early detection of AKI increases the likelihood of AKI prevention, associated morbidity, and costs [5]. As no specific treatment can reverse AKI and the recognition of patients at risk of AKI before diagnosis contributes to better clinical outcomes than treatment after AKI occurs [6], early detection of AKI is essential for prompt therapeutic intervention.

Several studies have attempted to predict AKI occurrence. With the increasing availability of clinical databases, they developed models to predict the occurrence of AKI using electronic health records (EHRs) [7-18]. Although these studies used EHRs, the number of patients in the patient population was small because they focused on specific patients, such as surgical patients, patients with sepsis, and older adults. There have also been a number of studies using artificial intelligence (AI) models to predict AKI. Although attempts have been made to predict the occurrence of AKI early, few models have provided clear rationales and explanations [19-21]. Therefore, time series data analysis is required for AKI prediction models to reflect the temporal information between variables [22]. Time series analysis for AKI is necessary because the length of time each patient stays in a hospital or intensive care unit can differ from person to person, and the frequency of measurements can vary from values that are measured continuously (eg, blood pressure) to laboratory values that are measured on an as-needed basis. Recently, an interpretable multivariable long short-term memory (IMV-LSTM) method for considering time series data has been published [23]; however, little research has been conducted on this method.

To address these issues, this study aimed to apply and validate a multicenter-based explainable time series AI model for predicting the occurrence of AKI caused by specific nephrotoxic drugs in 6 hospitals in South Korea by using a large clinical database with a common data model (CDM) through a distributed research network (DRN).

Ethical Considerations

This study was approved by the institutional review committees of Severance Hospital (SH; 4-2021-1209), Gangnam Severance Hospital (GSH; 3-2021-0005), Konyang University Hospital (KYUH; 2021-10-003-001), Ajou University Hospital (AJUH; AJIRB-MED-MDB-21‐676), Seoul National University Cancer Hospital (SNUH; E-2207-151-1342), and the National Cancer Center (NCC; 2022-0184). All retrospective data were anonymized and appropriate measures were taken to protect participant information.

Study Design

This retrospective observational cohort study analyzed the EHRs from 6 hospitals in South Korea between 1994 and 2021 to predict AKI. The EHRs were converted to OMOP-CDM (Observational Medical Outcomes Partnership Common Data Model) version 5.3.1. The 6 hospitals were SH, GSH, KYUH, AJUH, SNUH, and the NCC. An overall diagram of the cohort composition is provided in Figure 1.

In our cohorts, we adopted criteria based on serum creatinine (SCr) to define AKI according to the Kidney Disease: Improving Global Outcomes (KDIGO) Clinical Practice Guidelines and the previously defined AKI classification stages mapped in the “injury” category [24-26]. The criterion is an increase in the SCr level to 2 times the baseline value. As an alternative to the baseline SCr levels, we defined the upper limit of normal (ULN) value of SCr as 1.2 mg/dl [27].

The targeted drugs were selected from 5 medications associated with a high risk of AKI according to the US Food and Drug Administration (FDA) and previous studies: acetaminophen; vancomycin; 2 nonsteroidal anti-inflammatory drugs (NSAIDs), naproxen and celecoxib; and 1 antiviral drug, acyclovir [21,22].

The inclusion criteria were as follows: (1) the target drugs were administered, (2) the patient had a visit record of at least 30 days prior to the observation period, and (3) the patient underwent at least 2 SCr tests during the preobservation period (0‐60 days before the study). The exclusion criteria were as follows: (1) the patient had at least 1 SCr test outside the ULN value in the preobservation period. Participants were divided into case and control cohorts based on whether they met previously defined AKI criteria for 60 days after taking the first medicine. The observation period refers to the time range before and after the initial medication intake for each patient within the cohort. The cohort definitions were created using ATLAS, a web-based tool (Observational Health Data Sciences and Informatics), and are available as JSON files on GitHub [28].

To adjust for differences between cases and controls to reduce the effect of confounding variables, we used propensity score matching (PSM). Covariates included were age, sex, and SCr value at baseline. We normalized the covariates by applying standard scaling to ensure consistent dimensions across variables. A propensity score for each patient was generated by logistic regression. Patients were matched in a 1:3 ratio using a K-nearest neighbor (K-NN) algorithm using the Python scikit-learn library.

Figure 1. Overall flowchart for predicting acute kidney injury events. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital; IMV-LSTM: interpretable multivariable long short-term memory; KYUH: Konyang University Hospital; NCC: National Cancer Center; OMOP: Observational Medical Outcomes Partnership; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital; ULN, upper limit of normal.

Candidate Predictors for Time Series

Candidate predictors were extracted from several key domains within the CDM containing per-patient observational data using structured query language tools in Python. Age and sex were used in the person domain, clinical laboratory tests in the measurement domain, medications in the drug exposure domain, diagnostic records in the condition occurrence domain, and surgical/procedure records in the procedure occurrence domain. Lab tests were treated as continuous variables, while other medications, conditions, and procedures were treated as binary variables. Statistical methods were used to select the variables. To identify predictors, we tested the statistical significance of the difference between the enrollment time of the cohort and the onset date of AKI using a 2-tailed paired t test and the McNemar test for continuous and dichotomous variables, respectively. To create a time series table, the candidate variables were pivoted into columns and dates were placed into rows. Missing values were handled in the following ways: forward fill for laboratory tests and diagnoses and zero fill for medications and treatments. The window size for predictions used 4-week sequence data and was processed by shifting the data of the prediction cycle by 2 weeks.

AKI Prediction Modeling

LSTM models based on recurrence have been designed to process time series data [29]. Attention-based LSTM models were initially proposed for learning words and the relationships between words in natural language processing [30,31] and later evolved into a key component of deep learning, becoming one of the methods used to provide interpretations, including importance scores for predicted outcomes.

As an advanced LSTM model, we used the IMV-LSTM module for the learning model, which is a multivariate LSTM neural network for the prediction and interpretation of multivariate time series [23]. This model improves on the LSTM-attention model, which can predict variable importance using multivariate inputs to configure variable-wise hidden states and mix both temporal and variable levels of attention for improved interpretability. The model was trained for 200 epochs with a batch size of 64 and a learning rate of 1e-3. An Adam optimizer was used with early stopping after 20 epochs. The data set was divided into training, test, and validation sets at a 6:2:2 ratio. Prediction performance was evaluated using the area under the receiver operating characteristic curve (AUROC) value. Additionally, we used the accuracy, precision, F1-score, and area under the precision-recall curve (AUPRC) to ensure robustness for unbalanced data.

In this study, AKI prediction models were created for each hospital and drug. Each model had a different selection of candidate variables. To interpret the predictors in each model, variable- and temporal-wise attention scores were extracted from all trained models. These scores were then aggregated by calculating the overall temporal attention score, which was obtained by taking the weighted average of the temporal attention value over the attention value for each predictor variable. The resulting scores were plotted as heat maps for interpretation.

Statistical Analysis

We used statistical packages based on Python and R (R Project for Statistical Computing) for the statistical analysis. First, to compare the AKI and non-AKI groups, we calculated significance using the χ2 test for categorical variables and an independent-sample 2-tailed t test for continuous variables. Second, to identify differences in the pattern of AKI occurrence between cohorts and drugs, a histogram was plotted for patients in each cohort from the date of cohort entry (the first day of administration of the target drug) to the date of AKI occurrence. Differences between drugs were analyzed using an independent-sample2-tailed t test. Third, we compared the distribution of the aggregated temporal attention scores with the actual trained data with box plots of the data for 4 weeks at 1-week intervals. A repeated ANOVA test was performed to identify temporal differences.

Demographic and Clinical Characteristics

The demographics of the 31,012 patients without AKI and the 8643 patients with AKI across the 6 hospitals after PSM are shown in Table 1.

Table 1. Demonstration and clinical characteristics of patients with and without AKI across 6 hospitals after propensity score matching. The P values were obtained by conducting a 2-sample t test to compare the means for cases and controls.
Case groupControl groupP value
SHa (n=3028)GSHb (n=491)KYUHc (n=539)AJUHd (n=1003)SNUHe (n=2616)NCCf (n=966)Total (n=8643)SH (n=11,018)GSH (n=1689)KYUH (n=1954)AJOU (n=3328)SNUH (n=9558)NCC (n=3465)Total (n=31,012)
Age (years), mean (SD)61.83 (15.23)62.6 (15.04)67.91 (13.64)59.52 (15.79)57.89 (16.4960.55 (12.24)60.65 (15.5)61.14 (15.37)62.26 (14.81)67.71 (13.55)59.67 (15.77)57.46 (16.61)60.14 (12.71)60.21 (15.61).02
Gender, n (%)
Male1965 (64.89)286 (58.25)360 (66.79)628 (62.61)1652 (63.15)569 (58.9)5460 (63.17)7092 (64.37)985 (58.32)950 (48.62)1765 (53.03)4602 (48.15)2018 (58.24)17,412 (56.15)<.001
Female1063 (35.11)205 (41.75)179 (33.21)375 (37.39)964 (36.85)397 (41.1)3183 (36.83)3926 (35.63)704 (41.68)1004 (51.38)1563 (46.97)4956 (51.85)1447 (41.76)13,600 (43.85)<.001
Sepsis, n (%)325 (10.73)50 (10.18)7 (1.3)139 (13.86)40 (1.53)1 (0.1)562 (6.5)373 (3.39)37 (2.19)21 (1.07)128 (3.85)41 (0.43)1 (0.03)601 (1.94)<.001
Diabetes mellitus, n (%)915 (30.22)117 (23.83)65 (12.06)223 (22.23)300 (11.47)59 (6.11)1679 (19.43)2462 (22.35)266 (15.75)195 (9.98)593 (17.82)836 (8.75)167 (4.82)4519 (14.57)<.001
Chronic kidney disease, n (%)142 (4.69)3 (0.61)24 (4.45)13 (1.3)18 (0.69)2 (0.21)202 (2.34)271 (2.46)8 (0.47)51 (2.61)11 (0.33)40 (0.42)3 (0.09)384 (1.24)<.001
Chronic liver disease, n (%)462 (15.26)56 (11.41)52 (9.65)139 (13.86)506 (19.34)16 (1.66)1231 (14.24)766 (6.95)82 (4.85)71 (3.63)126 (3.79)759 (7.94)42 (1.21)1846 (5.95)<.001
Hypoalbuminemia, n (%)7 (0.23)2 (0.41)00009 (0.1)9 (0.08)2 (0.12)000011 (0.04).01
Hypotension, n (%)12 (0.4)4 (0.81)01 (0.1)4 (0.15)021 (0.24)30 (0.27)3 (0.18)04 (0.12)28 (0.29)065 (0.21).56
Hypertension, n (%)1462 (48.28)179 (36.46)77 (14.29)351 (35)264 (10.09)71 (7.35)2404 (27.81)4178 (37.92)455 (26.94)360 (18.42)1143 (34.34)924 (9.67)229 (6.61)7289 (23.5)<.001
Neoplasm (active cancers), n (%)2112 (69.75)320 (65.17)255 (47.31)652 (65)2081 (79.55)581 (60.14)6001 (69.43)5512 (50.03)531 (31.44)417 (21.34)1282 (38.52)4450 (46.56)2470 (71.28)14,662 (47.28)<.001
Heart failure, n (%)266 (8.78)13 (2.65)34 (6.31)33 (3.29)62 (2.37)6 (0.62)414 (4.79)629 (5.71)20 (1.18)105 (5.37)60 (1.8)127 (1.33)10 (0.29)951 (3.07)<.001
Obesity, n (%)3 (0.1)2 (0.41)02 (0.2)3 (0.11)010 (0.12)36 (0.33)3 (0.18)4 (0.2)10 (0.3)50 (0.52)0103 (0.33)<.001
Peripheral vascular disease, n (%)25 (0.83)9 (1.83)19 (3.53)18 (1.79)10 (0.38)081 (0.94)51 (0.46)26 (1.54)69 (3.53)35 (1.05)37 (0.39)2 (0.06)220 (0.71).03
Liver dysfunction, n (%)62 (2.05)2 (0.41)2 (0.37)40 (3.99)31 (1.19)2 (0.21)139 (1.61)71 (0.64)4 (0.24)25 (1.28)46 (1.38)59 (0.62)3 (0.09)208 (0.67)<.001
Anemia, n (%)541 (17.87)25 (5.09)35 (6.49)89 (8.87)154 (5.89)4 (0.41)848 (9.81)903 (8.2)53 (3.14)68 (3.48)176 (5.29)282 (2.95)7 (0.2)1489 (4.8)<.001
Prior kidney surgery, n (%)001 (0.19)3 (0.3)14 (0.54)018 (0.21)000011 (0.12)1 (0.03)12 (0.04)<.001
Laboratory values (before medication), mean (SD)
Serum creatinine (mg/dL)0.46 (0.74)0.79 (0.23)0.99 (0.37)0.87 (0.37)0.87 (0.47)1.01 (0.29)0.75 (0.59)0.44 (0.51)0.76 (0.19)0.98 (0.22)0.84 (0.2)0.83 (0.18)0.98 (0.16)0.72 (0.4)<.001
Glucose (mg/dL)126.89 (58.33)132.1 (52.11)150.75 (82.36)187.0 (23.0)121.48 (48.26)125.85 (50.05)126.93 (56.28)121.72 (55.7)124.3 (49.43)140.11 (78.81)151.5 (39.5)115.65 (40.5)120.88 (46.23)120.95 (51.85)<.001
Potassium (mmol/L)3.92 (0.6)4.21 (0.47)4.14 (0.6)4.17 (0.55)4.22 (0.5)4.29 (0.56)4.11 (0.57)3.93 (0.55)4.22 (0.4)4.09 (0.54)4.15 (0.46)4.22 (0.43)4.28 (0.52)4.11 (0.51).48
Sodium (mmol/L)138.6 (4.16)137.27 (4.36)136.49 (4.61)138.48 (4.23)138.46 (3.96)138.31 (3.67)138.3 (4.14)139.44 (3.75)138.31 (3.58)137.74 (3.73)139.27 (3.85)139.71 (3.23)138.96 (3.37)139.28 (3.6)<.001
BUN (blood urea nitrogen) (mg/dL)17.83 (9.66)15.44 (6.58)17.67 (8.87)14.68 (6.36)14.72 (7.61)15.55 (6.13)16.13 (8.28)16.82 (7.48)15.78 (5.86)17.19 (7.62)14.9 (6.41)14.67 (5.14)14.63 (4.96)15.67 (6.47)<.001

aSH: Severance Hospital.

bGSH: Gangnam Severance Hospital.

cKYUH: Konyang University Hospital.

dAJUH: Ajou University Hospital.

eSNUH: Seoul National University Cancer Hospital.

fNCC: National Cancer Center.

By hospital, the cohort consisted of 14,046 patients from SH, 2180 from GSH, 2493 from KYUH, 4331 from AJUH, 13,174 from SNUH, and 4431 from NCC after matching the propensity scores. Propensity matching was performed using age, sex, and SCr levels at baseline. As for the changes in covariates, the difference in mean age decreased from 6.39 (60.65 – 54.26) to 0.44 (60.65 – 60.21) years, the difference in male ratio decreased from 16.49% (63.17% – 46.68%) to 7.02% (63.17% – 56.15%), and the difference in SCr at baseline decreased from 0.14 mg/dL (0.71 – 0.58 mg/dL) to 0.03 mg/dL (0.71 – 0.67 mg/dL) (Table S2 in Multimedia Appendix 1). There were still statistically significant differences in PSM age (60.66, SD 15.86 vs 60.22, SD 15.94 years; P=.03), gender (63.17% vs 56.15% male; P<.001), and SCr at baseline (0.71, SD 0.61 vs 0.68, SD 0.41 mg/dL; P<.001). Patients who developed AKI had more severe neoplasms (ie, active cancers; 70.6% vs 44.26%; P<.001) and chronic liver disease (15.83% vs 6.55%; P<.001). Moreover, the analysis revealed the following differences: sepsis (6.5% vs 1.94%; P<.001), diabetes mellitus (19.43% vs 14.57%; P<.001), hypertension (27.81% vs 23.5%; P<.001), anemia (9.81% vs 4.8%; P<.001), and heart failure (4.79% vs 3.07%; P<.001). There was no significant difference between hypotension (0.24% vs 0.21%; P=.56), potassium (4.11, SD 0.57 vs 4.11, SD 0.51 mmol/L; P=.48), or renal artery stenosis (0.07% vs 0.03%; P=.13). Hypoalbuminemia, obesity, peripheral vascular disease, renal artery stenosis, liver dysfunction, and prior kidney surgery had low incidence rates (<2%).

Distribution of Adverse Drug Events

To analyze the differences in drug patterns at the time of AKI occurrence, we assessed the pattern for each drug. The median number of days for the occurrence of AKI among the drug and cohort patients in the entire hospital was 17 (IQR 7-33 days). Vancomycin appeared after a median period of 12 days, followed by naproxen (18 days), acetaminophen (19 days), celecoxib (22 days), and acyclovir (23 days). When comparing the IQR values, celecoxib (10‐41 days) and Acyclovir (10‐41 days) showed a relatively broad distribution, whereas acetaminophen (9‐34 days) and naproxen (8‐34 days) were distributed over 25 days. Vancomycin (5‐25 days) exhibited the narrowest distribution. Celecoxib and acyclovir tended to be relatively distributed compared to acetaminophen, vancomycin, and naproxen. We compared the onset times of all drug pairs and hospital pairs to check the similarity in AKI occurrence (Figure 2). The patterns between specific drugs was similar for celecoxib and acyclovir (P=.88) and for acetaminophen and naproxen (P=.57). The patterns between hospitals were similar for SH and AJUH (P=.98), SH and GSH (P=.36), GSH and AJUH (P=.42), GSH and NCC (P=.24), and SNUH and NCC (P=.26).

Figure 2. (A) Comparison of acute kidney injury (AKI) onset time between drugs and (B) AKI onset time between hospitals. The P values were obtained by conducting independent 2-tailed t tests between each aggregated pair. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital; KYUH: Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital.

AKI Prediction Model Performance

The AUROC for each drug and hospital to evaluate the AKI predictive model, based on respective test sets (internal validation) is shown in Figure 3. A total of 26 trained models achieved a high AUROC value, of 0.92 on average, with each verification data set. In addition, among the averages of the drugs, acyclovir had the highest average AUROC score of 0.94, followed by acetaminophen (0.93), vancomycin (0.92), naproxen (0.90), and celecoxib (0.89). The highest AUROC value (0.97) was observed for the model of SH’s celecoxib and acyclovir, SNUH’s vancomycin, and KYUH’s acyclovir prescription patients.

Multimedia Appendix 2 presents data on the precision, accuracy, F1-score, and AUPRC of each predictive model. Overall, the average accuracy of the AKI prediction models was 0.88, whereas the average AUPRC and F1-scores were both 0.78. The acyclovir prescription model achieved the highest accuracy score (0.91), followed by vancomycin (0.90), acetaminophen (0.89), naproxen (0.89), and celecoxib (0.86). Individually, the acyclovir SH model showed the best performance, with an AUPRC of 0.92 and an accuracy of 0.91.

Figure 3. Receiver operating characteristic (ROC) curves and areas under the curve (AUCs) of the acute kidney injury prediction model for each hospital and each drug. The square brackets indicate the 95% CI. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital; KYUH: Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital.

Temporal Feature Importance of the AKI Prediction Model

To interpret the AKI prediction model, we demonstrated the temporal attention values of each contributing variable in the 4 weeks prior to AKI onset, which were weighted aggregates from the model for each drug and hospital, as shown in Figure 4A. The temporal change pattern of the actual data corresponding to each variable in the 4 weeks prior to AKI onset is shown in Figure 4B. We also confirmed the difference in the distribution of highly important features between the case and control data using a 1-way ANOVA. The attention scores for all variables across all hospitals are detailed in Multimedia Appendix 3

The last week of lymphocytes (attention score at −1 week: 0.41) and the second week of calcium (attention score at −3 weeks: 0.41) showed the highest attention scores, followed by albumin (attention score at −1 week: 0.37; attention score at −4 weeks: 0.37), hemoglobin (attention score at −4 weeks: 0.37), and cholesterol (attention score at −4 weeks: 0.37). In Figure 5, the distribution of data by variable between the 2 groups was confirmed using actual data. There was a difference in data distribution between the case and control groups from the beginning. The values of lymphocytes, albumin, and hemoglobin in the case group decreased over time, while urine pH and prothrombin time in the case group tended to increase over time.

Figure 4. (A) Temporal attention score of important features of acute kidney injury (AKI) prediction model and (B) distribution of data over time (P values: repeated measures ANOVA). The figure shows the change over a 4-week period prior to the AKI event.
Figure 5. Acute kidney injury (AKI) onset time after drug administration at various medical centers. The red lines shows the median value. AJUH’s naproxen and GSH’s acyclovir cases were excluded as the number of AKI case groups was less than 20. AJUH: Ajou University Hospital; GSH: Gangnam Severance Hospital; KYUH: Konyang University Hospital; NCC: National Cancer Center; SH: Severance Hospital; SNUH: Seoul National University Cancer Hospital.

Principal Findings

In this study, we developed time series–based IMV-LSTM models to predict AKI in patients taking specific nephrotoxic drugs using CDM-based DRNs in a 6-hospital EHR-based system. The principal findings are as follows: first, this study provides an interpretation of the temporal importance of variables for predicting AKI, and the models also achieved high performance, with an average AUC of 0.92%. Second, our study is a scalable multicenter study using a DRN, which can contribute to understanding drug-induced AKI. To the best of our knowledge, this is the first study to build an AKI prediction model by applying a time series–based IMV-LSTM model to a CDM using EHR data from 6 hospitals.

We established a retrospective cohort of patients who took nephrotoxicity-inducing drugs at 6 hospitals. With respect to demographic characteristics, we observed variations in the overall patient count and prevalence of comorbidities when comparing individuals with AKI and without AKI across different hospitals. Nevertheless, the majority of patients who developed AKI at most hospitals were older than 60 years and had a high prevalence of comorbidities, including cancer (n=6001, 69.43%), hypertension (n=2404, 27.81%), diabetes (n=1679, 19.43%), and chronic liver disease (n=1231, 14.24%), which is consistent with findings reported in previous studies [32-35].

The pattern of each drug’s association with AKI (Figure 5) showed that the median number of days for AKI onset when using nephrotoxic drugs was 17 (IQR 7-33) days. The onset occurred earliest with vancomycin (12, IQR 5-25 days) and latest with acyclovir (23, IQR 10-41 days). In previous studies [36,37], the time to onset of vancomycin-induced AKI showed a similar pattern to our results. We also found differences in the AKI onset between different classes within the same NSAID, and the multicenter AKI cohort showed similarities between hospitals. The finding of similar patterns in the AKI onset in the multicenter cohort supports the reliability of the AKI cohort and increases the explanatory power of AKI prediction models.

AKI is common among inpatients [38,39]. Previous models predicted AKI in the intensive care unit (ICU) and in surgical patients during hospital admission. For example, Zimmerman et al [40] predicted the occurrence of AKI in ICU inpatients (AUC 0.783), and Tseng et al [41] developed a predictive score for the development of AKI after cardiac surgery (AUC 0.839). Hsu et al [42] developed a risk score function for community-acquired AKI for inpatients (AUC 0.818). Koyner et al [10] developed a model to predict AKI in hospitalized patients (AUC 0.90). Despite this progress, few studies have applied time series deep learning to provide a temporal interpretation of drug-induced AKI, and our model stands out because it can predict nephrotoxic drug–induced AKI in a diverse hospital population.

This study achieved improved performance compared to previous AKI studies using recurrent neural networks (RNNs) [21,43-45]. Our model improves performance up to an AUC of 0.97 and an overall average of 0.92, which outperforms previous studies showing AKI prediction with RNN-based methods by Kim et al [43] in hospitalized patients (AUC 0.927), Rank et al [45] in cardiac surgery patients (AUC 0.893), and Xu et al [44] in inpatients (AUC 0.908). These results show promise for our model as a tool to predict AKI and facilitate early intervention and mitigation strategies for patients.

This study also provides additional interpretations regarding the temporal importance of features for AKI prediction. Some studies have reported information on interpretability or provided information about the interpretability of variables at the feature importance level [46]. However, our results show the importance of variables and the temporal importance of variables in the development of AKI. In this study, we highlight the vital role of temporal patterns of various indicators, such as lymphocytes, calcium, albumin, hemoglobin, and cholesterol, in predicting disease states, particularly the onset of AKI. The temporal pattern of lymphocytes increased gradually, peaking 1 week before AKI onset. The use of lymphocyte and neutrophil counts as predictive factors for AKI is consistent with other studies [47,48]. Calcium shows a pattern of peaking 3 weeks before AKI onset, and Prior studies showed an association between impacted calcium metabolism and AKI [49,50]. Albumin shows the highest pattern 1 and 4 weeks before onset, and low serum albumin levels (hypoalbuminemia) are a predictor of AKI [48,51,52]. Hemoglobin shows a pattern with a peak 4 weeks before onset, and previous studies have shown that the risk of AKI increases stepwise with a further decrease in hemoglobin concentration [53]. Temporal variations in variables based on reported laboratory data for the early detection of AKI emphasize the importance of monitoring and early intervention in populations.

In addition, this retrospective study can be followed by a subsequent study to validate the practicality of the AKI prediction model in clinical practice by applying it to a prediction system in a hospital EHR.


This study has several limitations. First, because we used the CDM, it does not reflect the full range of clinical data. For example, we could not include admission records, which would have revealed a patient’s condition. However, the use of CDM data allowed for a multicenter study that could be easily extended to other institutions that have converted to the CDM. Second, this was a retrospective study and could not address the underlying causes of AKI. Therefore, prospective studies are needed for validation with actual clinical data. Third, as with all retrospective studies, there may be unintentional patient selection bias and unaccounted-for confounders. However, to compensate for these limitations, we tried to equalize the distribution of patient characteristics through PSM. We also used a limited follow-up period to minimize the impact of these factors.


This study demonstrates the high performance of the IMV-LSTM method for AKI prediction using hospital EHR-based time series data. Our model can provide real-time assessment of AKI occurrence and individualized risk factors for AKI using time series data. We also demonstrated the robustness of our model through multicenter validation using a CDM through a DRN of 6 hospitals in South Korea, which also proves that scalability to other institutions that are converted to the CDM is possible. This may provide an objective quantitative tool for identifying patients at risk of developing AKI.


This study was supported by a grant from the Korea Institute of Drug Safety and Risk Management in 2021. This work was supported by the Bio-Industrial Technology Development Program (20014841) funded by the Ministry of Trade, Industry & Energy (Korea). We would like to thank the Medical Informatics Collaborative Unit members of Yonsei University College of Medicine for their assistance in data analysis.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Demographics for each institution.

XLSX File, 60 KB

Multimedia Appendix 2

Model performance.

XLSX File, 10 KB

Multimedia Appendix 3

Feature importance.

XLSX File, 67 KB

  1. Chertow GM, Burdick E, Honour M, Bonventre JV, Bates DW. Acute kidney injury, mortality, length of stay, and costs in hospitalized patients. J Am Soc Nephrol. Nov 2005;16(11):3365-3370. [CrossRef] [Medline]
  2. Waikar SS, Curhan GC, Ayanian JZ, Chertow GM. Race and mortality after acute renal failure. J Am Soc Nephrol. Oct 2007;18(10):2740-2748. [CrossRef] [Medline]
  3. Kee YK, Kim EJ, Park KS, et al. The effect of specialized continuous renal replacement therapy team in acute kidney injury patients treatment. Yonsei Med J. May 2015;56(3):658-665. [CrossRef] [Medline]
  4. Medina KRP, Jeong JC, Ryu JW, et al. Comparison of outcomes of mild and severe community- and hospital-acquired acute kidney injury. Yonsei Med J. Oct 2022;63(10):902-907. [CrossRef] [Medline]
  5. Kate RJ, Perez RM, Mazumdar D, Pasupathy KS, Nilakantan V. Prediction and detection models for acute kidney injury in hospitalized older adults. BMC Med Inform Decis Mak. Mar 29, 2016;16:39. [CrossRef] [Medline]
  6. Khwaja A. KDIGO Clinical Practice Guidelines for acute kidney injury. Nephron Clin Pract. 2012;120(4):c179-c184. [CrossRef] [Medline]
  7. Ugwuowo U, Yamamoto Y, Arora T, et al. Real-time prediction of acute kidney injury in hospitalized adults: implementation and proof of concept. Am J Kidney Dis. Dec 2020;76(6):806-814. [CrossRef] [Medline]
  8. Low S, Vathsala A, Murali TM, et al. Electronic health records accurately predict renal replacement therapy in acute kidney injury. BMC Nephrol. Jan 31, 2019;20(1):32. [CrossRef] [Medline]
  9. Chua HR, Zheng K, Vathsala A, et al. Health care analytics with time-invariant and time-variant feature importance to predict hospital-acquired acute kidney injury: observational longitudinal study. J Med Internet Res. Dec 24, 2021;23(12):e30805. [CrossRef] [Medline]
  10. Koyner JL, Carey KA, Edelson DP, Churpek MM. The development of a machine learning inpatient acute kidney injury prediction model. Crit Care Med. Jul 2018;46(7):1070-1077. [CrossRef] [Medline]
  11. Wilson FP, Greenberg JH. Acute kidney injury in real time: prediction, alerts, and clinical decision support. Nephron. 2018;140(2):116-119. [CrossRef] [Medline]
  12. Huang CY, Güiza F, De Vlieger G, et al. Development and validation of clinical prediction models for acute kidney injury recovery at hospital discharge in critically ill adults. J Clin Monit Comput. Feb 2023;37(1):113-125. [CrossRef] [Medline]
  13. Bedford M, Stevens P, Coulton S, et al. Development of risk models for the prediction of new or worsening acute kidney injury on or during hospital admission: a cohort and nested study. Health Serv Deliv Res. 2016;4(6):1-160. [CrossRef] [Medline]
  14. Kim WH, Lee SM, Choi JW, et al. Simplified clinical risk score to predict acute kidney injury after aortic surgery. J Cardiothorac Vasc Anesth. Dec 2013;27(6):1158-1166. [CrossRef] [Medline]
  15. Kiers HD, van den Boogaard M, Schoenmakers MCJ, et al. Comparison and clinical suitability of eight prediction models for cardiac surgery-related acute kidney injury. Nephrol Dial Transplant. Feb 2013;28(2):345-351. [CrossRef] [Medline]
  16. Zhou LZ, Yang XB, Guan Y, et al. Development and validation of a risk score for prediction of acute kidney injury in patients with acute decompensated heart failure: a prospective cohort study in China. J Am Heart Assoc. Nov 16, 2016;5(11):e004035. [CrossRef] [Medline]
  17. Wilson T, Quan S, Cheema K, et al. Risk prediction models for acute kidney injury following major noncardiac surgery: systematic review. Nephrol Dial Transplant. Feb 2016;31(2):231-240. [CrossRef] [Medline]
  18. Sanchez-Pinto LN, Khemani RG. Development of a prediction model of early acute kidney injury in critically ill children using electronic health record data. Pediatr Crit Care Med. Jun 2016;17(6):508-515. [CrossRef] [Medline]
  19. Sutherland SM, Chawla LS, Kane-Gill SL, et al. Utilizing electronic health records to predict acute kidney injury risk and outcomes: workgroup statements from the 15th ADQI Consensus Conference. Can J Kidney Health Dis. 2016;3:11. [CrossRef] [Medline]
  20. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. May 2019;1(5):206-215. [CrossRef] [Medline]
  21. Dong J, Feng T, Thapa-Chhetry B, et al. Machine learning model for early prediction of acute kidney injury (AKI) in pediatric critical care. Crit Care. Aug 10, 2021;25(1):288. [CrossRef] [Medline]
  22. Lee JM, Hauskrecht M. Modeling multivariate clinical event time-series with recurrent temporal mechanisms. Artif Intell Med. Feb 2021;112:102021. [CrossRef] [Medline]
  23. Guo T, Lin T, Antulov-Fantulin N, editors. Exploring interpretable LSTM neural networks over multi-variable data. In: Proceedings of the 36th International Conference on Machine Learning. PMLR; 2019.
  24. Kellum JA, Lameire N, Aspelin P, et al. Kidney disease: improving global outcomes (KDIGO) acute kidney injury work group. KDIGO Clinical Practice Guideline for acute kidney injury. Kidney Int Suppl. 2012;2(1):1-138. [CrossRef]
  25. Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, Acute Dialysis Quality Initiative workgroup. Acute renal failure - definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care. Aug 2004;8(4):R204-R212. [CrossRef] [Medline]
  26. Mehta RL, Kellum JA, Shah SV, et al. Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Crit Care. 2007;11(2):R31. [CrossRef] [Medline]
  27. Hosten AO. BUN and creatinine. In: Clinical Methods: The History, Physical, and Laboratory Examinations. 3rd ed. Butterworths; 1990.
  28. Development and verification of a time series AI model for acute kidney injury detection based on a multicenter distributed research network. GitHub. URL: [Accessed 2023-07-08]
  29. Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. Nov 15, 1997;9(8):1735-1780. [CrossRef] [Medline]
  30. Luong MT, Pham H, Manning CD. Effective approaches to attention-based neural machine translation. arXiv. Preprint posted online on Sep 20, 2015. [CrossRef]
  31. Bahdanau D, Cho K, Bengio Y. Neural machine translation by jointly learning to align and translate. arXiv. Preprint posted online on Sep 1, 2014. [CrossRef]
  32. Harirforoosh S, Asghar W, Jamali F. Adverse effects of nonsteroidal antiinflammatory drugs: an update of gastrointestinal, cardiovascular and renal complications. J Pharm Pharm Sci. 2013;16(5):821-847. [CrossRef] [Medline]
  33. Perazella MA. Drug-induced nephropathy: an update. Expert Opin Drug Saf. Jul 2005;4(4):689-706. [CrossRef] [Medline]
  34. Khan S, Loi V, Rosner MH. Drug-induced kidney injury in the elderly. Drugs Aging. Oct 2017;34(10):729-741. [CrossRef] [Medline]
  35. Finlay S, Bray B, Lewington AJ, et al. Identification of risk factors associated with acute kidney injury in patients admitted to acute medical units. Clin Med (Lond). Jun 2013;13(3):233-238. [CrossRef] [Medline]
  36. van Hal SJ, Paterson DL, Lodise TP. Systematic review and meta-analysis of vancomycin-induced nephrotoxicity associated with dosing schedules that maintain troughs between 15 and 20 milligrams per liter. Antimicrob Agents Chemother. Feb 2013;57(2):734-744. [CrossRef] [Medline]
  37. Filippone EJ, Kraft WK, Farber JL. The nephrotoxicity of vancomycin. Clin Pharmacol Ther. Sep 2017;102(3):459-469. [CrossRef] [Medline]
  38. Kate RJ, Pearce N, Mazumdar D, Nilakantan V. A continual prediction model for inpatient acute kidney injury. Comput Biol Med. Jan 2020;116:103580. [CrossRef] [Medline]
  39. Kellum JA, Romagnani P, Ashuntantang G, Ronco C, Zarbock A, Anders HJ. Acute kidney injury. Nat Rev Dis Primers. Jul 15, 2021;7(1):52. [CrossRef] [Medline]
  40. Zimmerman LP, Reyfman PA, Smith ADR, et al. Early prediction of acute kidney injury following ICU admission using a multivariate panel of physiological measurements. BMC Med Inform Decis Mak. Jan 31, 2019;19(Suppl 1):16. [CrossRef] [Medline]
  41. Tseng PY, Chen YT, Wang CH, et al. Prediction of the development of acute kidney injury following cardiac surgery by machine learning. Crit Care. Jul 31, 2020;24(1):478. [CrossRef] [Medline]
  42. Hsu CN, Liu CL, Tain YL, Kuo CY, Lin YC. Machine learning model for risk prediction of community-acquired acute kidney injury hospitalization from electronic health records: development and validation study. J Med Internet Res. Aug 4, 2020;22(8):e16903. [CrossRef] [Medline]
  43. Kim K, Yang H, Yi J, et al. Real-time clinical decision support based on recurrent neural networks for in-hospital acute kidney injury: external validation and model interpretation. J Med Internet Res. Apr 16, 2021;23(4):e24120. [CrossRef] [Medline]
  44. Xu J, Hu Y, Liu H, et al. A novel multivariable time series prediction model for acute kidney injury in general hospitalization. Int J Med Inform. May 2022;161:104729. [CrossRef] [Medline]
  45. Rank N, Pfahringer B, Kempfert J, et al. Deep-learning-based real-time prediction of acute kidney injury outperforms human predictive performance. NPJ Digit Med. 2020;3:139. [CrossRef] [Medline]
  46. Wang Y, Wei Y, Yang H, Li J, Zhou Y, Wu Q. Utilizing imbalanced electronic health records to predict acute kidney injury by ensemble learning and time series model. BMC Med Inform Decis Mak. Sep 21, 2020;20(1):238. [CrossRef] [Medline]
  47. Weller S, Varrier M, Ostermann M. Lymphocyte function in human acute kidney injury. Nephron. 2017;137(4):287-293. [CrossRef] [Medline]
  48. Ishikawa M, Iwasaki M, Namizato D, et al. The neutrophil to lymphocyte ratio and serum albumin as predictors of acute kidney injury after coronary artery bypass grafting. Sci Rep. Sep 14, 2022;12(1):15438. [CrossRef] [Medline]
  49. Wang B, Li D, Gong Y, Ying B, Cheng B. Association of serum total and ionized calcium with all-cause mortality in critically ill patients with acute kidney injury. Clinica Chimica Acta. Jul 2019;494:94-99. [CrossRef]
  50. Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Erickson SB. Admission calcium levels and risk of acute kidney injury in hospitalised patients. Int J Clin Pract. Apr 2018;72(4):e13057. [CrossRef] [Medline]
  51. Wiedermann CJ, Wiedermann W, Joannidis M. Hypoalbuminemia and acute kidney injury: a meta-analysis of observational clinical studies. Intensive Care Med. Oct 2010;36(10):1657-1665. [CrossRef] [Medline]
  52. Thongprayoon C, Cheungpasitporn W, Mao MA, Sakhuja A, Kashani K. U-shape association of serum albumin level and acute kidney injury risk in hospitalized patients. PLoS One. 2018;13(6):e0199153. [CrossRef] [Medline]
  53. Walsh M, Garg AX, Devereaux PJ, Argalious M, Honar H, Sessler DI. The association between perioperative hemoglobin and acute kidney injury in patients having noncardiac surgery. Anesth Analg. Oct 2013;117(4):924-931. [CrossRef] [Medline]

ADR: adverse drug reaction
AI: artificial intelligence
AJUH: Ajou University Hospital
AKI: acute kidney injury
AUPRC: area under the precision-recall curve
AUROC: area under the receiver operating characteristic curve
CDM: common data model
DRN: distributed research network
EHR: electronic health record
FDA: Food and Drug Administration
GSH: Gangnam Severance Hospital
ICU: intensive care unit
IMV-LSTM: interpretable multivariable long short-term memory
KDIGO: Kidney Disease Improving Global Outcomes
KYUH: Konyang University Hospital
MIMIC: Medical Information Mart for Intensive Care
NCC: National Cancer Center
NSAID: nonsteroidal anti-inflammatory drug
OMOP-CDM: Observational Medical Outcomes Partnership Common Data Model
PSM: propensity score matching
RNN: recurrent neural network
SCr: serum creatinine
SH: Severance Hospital
SNUH: Seoul National University Cancer Hospital
ULN: upper limit of normal

Edited by Christian Lovis; submitted 30.03.23; peer-reviewed by Marek Oja, Xiangzhou Zhang; final revised version received 08.07.23; accepted 19.05.24; published 05.07.24.


© Suncheol Heo, Eun-Ae Kang, Jae Yong Yu, Hae Reong Kim, Suehyun Lee, Kwangsoo Kim, Yul Hwangbo, Rae Woong Park, Hyunah Shin, Kyeongmin Ryu, Chungsoo Kim, Hyojung Jung, Yebin Chegal, Jae-Hyun Lee, Yu Rang Park. Originally published in JMIR Medical Informatics (, 5.7.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.