We conducted a retrospective cohort study. The data from July 2010 to March 2017 were collected from the DPC database. All the patients in the database were included to maximize the generalizability of the results. During the study period, 1569 hospitals contributed to the database. The patients in the database represented about 50% of all the acute-care inpatients in Japan [14].

The following data are included in the DPC database: age, sex, admission date, discharge date, diagnoses, and procedures (drugs, examinations, and surgical and nonsurgical treatments) for each patient. In the DPC database, comorbidities present at admission are clearly distinguished from complications arising after admission. All diagnoses were recorded using the International Statistical Classification of Diseases and Related Health Problems, 10th revision (ICD-10) codes. Procedure records were coded with Japanese conventional codes.

The DPC database also includes several severity indices, namely, the Killip classification for AMI [15,16], New York Heart Association classification for HF [17], Barthel index score for activities of daily living at admission [18], Japan Coma Scale of consciousness level at admission [19]; and age, dehydration, respiration, orientation, blood pressure (A-DROP), the Japan Respiratory Society community-acquired pneumonia severity index [20,21]. The Japan Coma Scale is used widely in Japan to measure impaired consciousness: a score of 0 indicates alert consciousness; single-digit scores (1, 2, 3) indicate being awake without stimuli; double-digit scores (10, 20, 30) indicate patients can be aroused by some stimuli; and triple-digit scores (100, 200, 300) indicate coma. A-DROP is a system for scoring severity of pneumonia that includes age (men ≥70 years, women ≥75 years), dehydration (serum urea nitrogen ≥21 mg/dL), respiratory failure (oxygen saturation by pulse oximetry ≤90% or PaO₂ ≤60 mm Hg), orientation disturbance (confusion), and low blood pressure (systolic blood pressure ≤90 mm Hg).

Our study was approved by the Ethics Committee of the University of Tokyo School of Medicine (approval number: 3501-(4)).

We extracted the data of inpatients who were discharged from hospitals between July 1, 2010, and March 31, 2017. The study population was divided randomly into a derivation cohort (95%) and a validation cohort (5%). For cases with 1-day hospitalization, the time at which we collected the information for prediction and the time at which the outcome occurred could be simultaneous. Because this could lead to an overestimation of the accuracy of the prediction model, we excluded patients who were discharged or died on the day of hospitalization from the validation cohort.

The outcome variable was in-hospital death. For predictive variables, we used patients’ demographic information (age, sex, and history of hospitalization in the 180 days before admission), all the ICD-10-based diagnoses at admission, and all the procedures performed on the day of admission. Age was handled as a continuous variable; the other variables were handled as dichotomous variables (0 or 1). We also extracted the Killip classification [15,16], New York Heart Association classification [17], Barthel index score [18], Japan Coma Scale [19], and the A-DROP score [20,21] as common severity indices for specific diseases from the DPC database.

We developed a deep neural network model as the main model for predicting in-hospital death for all the patients, using 9 layers with 4 hidden dense layers [22,23]. For this, we used the patients’ demographic information, all the ICD-10–based diagnoses at admission, and all the procedures performed on the day of admission. All the layers had 1000 nodes and were fully connected to adjacent layers. We used a softmax layer with 2 nodes as the output layer [24]. Because the numbers of deceased and alive patients were very different, we weighted the deceased cases with the reciprocal of the proportion of deceased cases (ie, 1/0.045=22.3) [23]. We used stochastic gradient descent to obtain neural network weights iteratively [25]. To avoid overfitting, 20% drop-out layers were sandwiched within each of the dense layers and an early stopping procedure involving learning steps using 3% data in the derivation cohort was employed [26]. Details of the weight optimization process are described in Multimedia Appendix 1.

We constructed disease-specific models for predicting in-hospital mortality in subgroups with AMI, HF, stoke, or pneumonia. The 4 models included patient backgrounds (age, sex, and history of hospitalization in the 180 days before admission) and diagnoses, and none of the models included procedures. For the AMI-specific model, we selected patients with AMI and included the Killip classification [15,16]. For the HF-specific model, we selected patients with HF and included the New York Heart Association classification [17]. For the stroke-specific model, we selected patients with stroke and included the Barthel index and the Japan Coma Scale at admission [18,19]. For the pneumonia-specific model, we selected patients with pneumonia and included the A-DROP score [20,21].

We applied the main model to the subgroups of patients with AMI, HF, stoke, and pneumonia and compared its prediction performance with the prediction performances of the disease-specific models for AMI, HF, stoke, and pneumonia. We evaluated the performance of each model by calculating performance measures in the validation cohort. Performance measures included the area under the receiver operating characteristic curve (AUC), used to determine the discriminatory ability of the model. We calculated the 95% CI of the AUC using the DeLong method [27] and plotted a calibration curve to determine goodness of fit. We also calculated sensitivity, specificity, and positive and negative predictive values at the threshold determined by the Youden Index method [28]. We obtained CIs for all the indices with 2000 bootstraps.

We also examined whether the risk scores calculated by the disease-specific models improved the discrimination ability of the risk scores calculated by the main model. We incorporated the risk scores calculated by the main and disease-specific models into predictor variables of a logistic regression model and calculated combined risk scores that predicted in-hospital mortality for each disease population. The discrimination ability of the combined risk score was evaluated by its AUC and compared with the AUC of the main model. CIs for the AUC and hypothesis testing for the difference between the main model risk score and combined risk score were calculated using the DeLong method.

We constructed deep leaning–based prediction models for in-hospital mortality using a large Japanese inpatient database. Patient backgrounds, diagnoses, and treatments on the first day of admission were entered into the models. The overall discriminant abilities of the models were high in subgroups of patients with AMI, HF, stroke, and pneumonia. The main model had better discriminant abilities than disease-specific models using common severity indices. We integrated the risk scores for the main and disease-specific models and calculated combined risk scores. However, the improvement in the predictive performance of the combined risk scores over that of the main model risk scores was only slight.

Risk scores derived from administrative claims databases have been developed previously. For example, the Charlson and Elixhauser models, which use comorbidity information to predict long-term survival, have been used for risk adjustment in clinical and epidemiological studies [30,31]. In this study, a new prediction model for in-hospital mortality developed using administrative claims data showed high discriminatory power (AUC=0.945). We believe that our model can also be used for risk adjustment in clinical and epidemiological studies using administrative claims data that includes diagnoses and procedures.

In a previous study, we constructed a prediction model for in-hospital mortality that incorporated comorbidities and several selected procedures (blood tests, radiography, echocardiogram) on the day of admission [7]. However, that model lacked generalizability; for example, it was not applicable to critically ill patients. The newly constructed model can be used for risk prediction and adjustment for patients with a wide range of disease severity.

In a previous study, the predictive abilities of models with administrative claims data alone were compared with those of models with electronic medical records combined with administrative claims data [32]. The predictive abilities of the models with electronic medical records were higher because the electronic medical records included detailed information related to each patient, such as blood test results, vital signs, and admission data collected during the first 2 days of the index admission.

In this study, a deep learning model that used only massive administrative data had higher predictive ability than models that used disease-specific severity information. On the basis of our results, we consider that large-scale administrative data can be used to predict in-hospital mortality more accurately than the generally used severity indices. Kharrazi et al [33] reported that obtaining information from both administrative data and electronic health records increased the prediction accuracy of their model compared with using each data source alone. Zeltzer et al [32] found that feeding the electronic health record information collected during hospitalization, in addition to the administrative data and pre-hospitalization electronic health record information, into their model resulted in more accurate mortality risk assessment. Rajkomar et al [9] predicted in-hospital mortality with the same level of accuracy as we achieved in this study by using information from structured electronic health records. We also found that a combined risk score, obtained by integrating the main model with a disease-specific model, showed higher prediction accuracy than the risk score obtained from the main model. However, in this study, the difference between the main model and the combined risk score was small, and there was no significant difference between the two risk scores for AMI. This indicates that the main model was able to construct a risk score comparable to the combined risk score even without disease-specific severity information. Therefore, we propose that patient outcome studies can be conducted using administrative data alone, such as the initial hospitalization process and diagnosis, without the need for data on disease severity.

It is not easy to collect electronic health record information in a standardized way and use it for research. We believe that the results of this study can be used in cases where it is not possible to obtain detailed clinical information, such as disease severity and vital signs, that would be included in an electronic health record.

This study has several limitations. First, we did not conduct an external validation. Second, we did not use a variety of machine learning methods (eg, random forest, lasso regression, XGBoost, and their ensembles), so we could not compare the prediction performance of other machine learning methods. Third, because the database used in this study is for acute hospitalization, we could not obtain data on long-term outcomes. Fourth, model accuracy is not always guaranteed for all diseases, so the applicability of the model to other populations needs to be considered.

In conclusion, we constructed a deep neural network model to predict in-hospital mortality using all the data on diagnoses and procedures performed on the day of admission in a Japanese administrative claims database. Our model using only administrative claims data showed higher prediction ability than our models using the more generally used severity indices. We propose that prognostic models using data on diagnoses and procedures obtained only from administrative claims databases can predict in-hospital mortality and can be used for risk adjustment in clinical and epidemiological studies.