We used retrospective datasets from the Health and Retirement Study (HRS) [16] of the United States for model training and internal testing, which were selected between the year of 1992 and 2020 and followed up every 2 years, composing a population of more than 20,000. The cohort of the China Health and Retirement Longitudinal Study (CHARLS) [17] was adopted for model external validation, which involved a population of over 17,000 aged 45 years and older from 2010 to 2020. Lung cancer incidence, lung cancer survival, and overall survival were involved in this research. The inclusion criteria were aged 50 years and older and having respiratory disease–related questionnaire responses in follow-up surveys. Those without complete follow-up data were excluded. Risk factors such as person number, birth year, gender, marital status, BMI, years of education, smoking status, physical activity, drinking status, preventive behaviors, cancer history, and lung disease condition were involved in risk prediction and transfer modeling. We used R console to extract and map variables between the HRS and CHARLS datasets. For example, in HRS, the variable “PN” was used to represent a person number, while in CHARLS, the variable “ID” was used to denote a person number, as shown in Table 1. For years of education, specific education years were listed in HRS, while education types were supplied in CHARLS. We matched “no formal education illiterate” in CHARLS and “0 year of education” in HRS, and other education types were also matched similarly. Smoking cessation duration was calculated through “respondent smoking ever” and “respondent smoking now”; for example, if a respondent reported smoking ever in survey wave 1 and reported not smoking now in survey wave 2, we then deduced the respondent’s smoking cessation duration. Preventive behavior for cholesterol was involved in HRS and missed in CHARLS, which had no significant correlation with this research. Therefore, we did not include such variable.

Lung cancer risk stratification was conducted based on the deep neural network (DNN) model and the risk transfer pathway was simulated using the DQN model. Lung cancer incidence, lung cancer survival, and overall survival were calculated to evaluate lung cancer risk stratification. Lung cancer incidence survival referred to the time from the initial follow-up to lung cancer diagnosis. Lung cancer survival indicated the interval between the initial follow-up and death from lung cancer or the last follow-up. Overall survival represented the time from the initial follow-up to death from any cause or the last follow-up. Four stratified risk degrees of lung cancer were generated using the weight thresholds of the DNN model, which included high risk, medium risk, low risk, and non-risk. Further, 3 risk transfer directions were deduced and analyzed through the DQN model, including direction of risk improvement, direction of risk deterioration, and risk maintenance (maintaining the original risk state). Risk improvement represented the risk decrease of lung cancer occurrence; risk deterioration reflected the risk increase of lung cancer occurrence; and risk maintenance indicated that the risk of lung cancer remained unchanged. The accuracy and the area under the receiver operating characteristic curve (AUROC) were computed to evaluate the performance of the DQN model. Risk transfer pathway for lung cancer was developed based on DQN models, as shown in Figure 1.

The risk assessment model of lung cancer occurrence was constructed based on the DNN method as shown in Figure 2. Because of the long follow-up time in HRS and CHARLS, it was inevitable that some indicators might have missing values. We adopted the multiple imputation method [18] to fill in the missing values. We used the mean value to perform initial imputation of missing values and used linear regression for iterative imputation. Then we updated the imputation value until realizing model convergence and reaching the maximum iteration. Therefore, missing values were predicted through multiple iterations. We also used singularly valuable decomposition [19] to reduce the white noises. We matched follow-up data according to each unique person number. Data imbalance problems were mitigated through the synthetic minority oversampling technique [20]. Datasets of HRS were randomly divided into the training set and the internal test set for the DNN-based risk assessment model, which included 8780 samples in the training set and 3763 samples in the internal test set. Further, 16,442 samples from CHARLS were adopted for external testing. The synthetic minority oversampling technique was used with the HRS training dataset to construct models. The k-nearest neighbor algorithm was applied to calculate the minority sample. The simulated samples were added to the HRS training dataset.

The preprocessed data were input to the DNN model’s input layer. We iteratively calculate the weight matrix of risk factors using Equation (1). Specifically, W_k and b_k were the weight matrix and the bias vector of the kth layer of the DNN model. We used Equation (1) to compute the output of the kth layer. The linear rectification function, as shown in Equation (2), was used as the activation function to improve representation capability of the DNN model. Through multiple iterations and calculations, we computed weight thresholds using Equation (3), which are shown in Figure 3. When the average weight of the sample was in the interval 0‐0.2 in the year of 1992, we defined it as non-risk. When the average weight was in the interval 0.2‐0.5, 0.5‐0.71, or 0.71‐0.93 in the year of 1992, we defined it as low risk, medium risk, or high risk, respectively. Similarly, risk stratifications of other follow-up years were obtained according to weight thresholds as shown in Figure 3. The DNN-based lung cancer risk assessment model’s outputs were risk stratification of lung cancer occurrence, including high risk, medium risk, low risk, and non-risk. The calibration plots of the DNN model with Brier score and log loss are shown in Multimedia Appendix 1, and the confusion matrix of the DNN model is listed in Multimedia Appendix 2. The random seed of the DNN model was 42. Person number, birth year, gender, marital status, BMI, years of education, smoking status, physical activity, drinking status, prevention behaviors, cancer history, and lung disease condition, and so on, were involved features. The initial learning rate was 0.01, the batch size was 32, and the initial episode was 30. Through multiple iterations, 3 layers and 22 nodes were derived from the hidden layer, 56 nodes of the input layer, and 1 node of the output layer. The activation function was the linear rectification function. The deepnet package and h2o framework were leveraged for DNN model construction.

DQN models were constructed through different risk groups, and internal and external testing were conducted for model validation. Datasets including 8780 samples from HRS were used for DQN model training. A total of 3763 samples from HRS were used as an internal test set and 16,442 samples from CHARLS were used as an external test set. DQN models were built through 4 risk groups. First, Q-learning and convolutional neural networks (CNNs) were combined to construct the DQN model. The stratified risk group was the model’s input. We used CNN to approximate the target value function (target Q function) of the transfer strategy as closely as possible. Target Q functions for different risk groups were obtained. Second, the DQN model was trained using Q-learning. We calculated the loss function through Equation (4). a was the transfer strategy corresponding to different cycles. s was the different risk states including high risk, medium risk, low risk, and non-risk. Q was the output value function, which represented the maximum cumulative transfer effect of transfer strategy a. Q(s,a; θ_i) was the output of the current network. Q_i was the output of the target network. Third, we adopted the stochastic gradient descent function to optimize the loss function. The target Q_i function of the transfer strategy was obtained using Equation (5), and the transfer strategies of different risk states were derived accordingly.

The state space of the DQN setup was the vector (high risk state, medium risk state, low risk state, and non-risk state), and the action space was the set of transfer strategies, such as extending the duration of smoking cessation, reducing the frequency of smoking, and so on. We constructed DQN models for the 4 risk groups separately. As shown in Figure 4, we took the low risk group as an example to illustrate state representation and transition processes. We simulated risk transition of the low risk group in multiple cycles through corresponding transfer strategies. Similarly, DQN models for the high risk group, medium risk group, and non-risk group were developed to explore risk transfer pathways of lung cancer. The reward function is described in Equation (6), which was the transfer effect of current network. We used the stochastic gradient descent function to optimize the loss function. Q(s’, a’; θ_i-1) was the output of the target network, Q(s,a; θ_i) was the output of the current network, r was the transfer effect of the current network, ε was the transfer environment, and γ was the discount factor between 0.9 and 1. The random seed of the DQN model was 50. Person number, birth year, gender, marital status, BMI, years of education, smoking status, physical activity, drinking status, prevention behaviors, cancer history, and lung disease condition, and so on, were involved features. The initial learning rate was 0.01, the batch size was 256, and the episode number was 500. One input layer with 32×32 neurons; 3 convolutional layers with 5×5, 4×4, 3×3 convolution kernels; and 1 output layer were leveraged for the DQN model. OpenAI Gym and PyTorch framework were used.

Transfer simulation of multiple pathways in different cycles for lung cancer occurrence was conducted in stratified risk groups through the DQN model. To explore optimal transfer strategy and transfer effect of different pathways, we calculated and deduced the optimum transfer pathway. The probabilities of lung cancer risk state transition for different risk groups were calculated for each transition cycle. Transfer pathways corresponding to various transfer directions, such as risk improvement, risk deterioration, and risk maintenance, were deduced. Then, the optimal transfer pathway and transfer effect were obtained through the DQN model.

Kaplan-Meier analysis was leveraged to demonstrate survival differences among the 4 stratified risk groups. We used a Cox proportional hazards model for the relative risk analysis between lung cancer incidence and smoking. R version 4.3.3 (R Core Team) was used for statistical analysis. The deepnet package and the h2o framework were used for DNN model training. OpenAI Gym and PyTorch framework were used for the DQN model. The survival and survminer packages were adopted for Kaplan-Meier analysis.

This study used secondary anonymized data from HRS and CHARLS, which were approved by the Institutional Review Board Committees of Michigan State University (HUM00061128) and the Biomedical Ethics Review Committee of Peking University (IRB00001052–11015). In this study, the individual data were anonymized; patients and the public were not involved in the design, conduct, and reporting stages of the research. All participants provided informed consent, and the study was conducted in accordance with the ethical principles outlined in the Declaration of Helsinki.

Baseline characteristics of the training set, internal test set, and external test set are listed in Table 2. A total of 12,543 samples from HRS were used for DQN model development, and 16,442 samples from CHARLS were used for external testing of the DQN model.

The risk stratification and the trend of risk transfer of lung cancer incidence in different follow-up years are shown in Figure 5, which displays the percentages of the high risk, medium risk, low risk, and non-risk groups. People in the high risk group had higher lung cancer incidence than the medium risk, low risk, and non-risk groups, as shown in Figure 6A and B. The horizontal axis represents the follow-up time, and the value 1.00 on the vertical axis indicates the absence of lung cancer. The value 0.75 represents that the incidence survival rate of lung cancer was 75%, and the corresponding incidence rate of lung cancer was 25%. Lung cancer incidence of the medium risk group was also higher than low risk and non-risk groups. At the last follow-up, lung cancer incidence of the high risk group was about 25%, which was much higher than the incidence of the low risk and non-risk groups. As shown in Figure 6C and D, the high risk group had lower overall survival than other risk groups. Moreover, lung cancer survival of the high risk group was also lower than medium risk, low risk, and non-risk groups, as shown in Figure 6E and F.

Furthermore, external testing of the DNN model was conducted using the CHARLS cohort, as shown in Figure 7. External test results and internal test results of lung cancer risk assessment based on DNN models were mainly consistent.

Lung cancer risk transfer strategy and pathway were explored through DQN models. We deduced different transfer pathways of lung cancer incidence for stratified risk groups, as shown in Figure 8. Risk improvement represented the reduced risk of lung cancer occurrence, including high risk to medium risk, then to low risk and non-risk; high risk to low risk, then to non-risk; high risk to non-risk; medium risk to low risk, then to non-risk; medium risk to non-risk; and low risk to non-risk. Risk deterioration reflected the increased risk of lung cancer occurrence, including non-risk to low risk, then to medium risk and high risk; non-risk to medium risk, then to high risk; non-risk to high risk; low risk to medium risk, then to high risk; low risk to high risk; medium risk to high risk; high risk to lung cancer occurrence; medium risk to lung cancer occurrence; low risk to lung cancer occurrence; and non-risk to lung cancer occurrence. Risk maintenance represented that the risk of lung cancer occurrence had not changed, including high risk, medium risk, low risk, and non-risk remaining unchanged separately. In the risk improvement pathways, extending the duration of smoking cessation—such as quitting smoking from less than 1 month to 1 year, quitting smoking from less than 1 month to 3 years, or quitting smoking from less than 1 month to more than 5 years—and reducing the frequency of smoking—such as from everyday smoking to someday smoking, from someday smoking to quitting smoking, or from everyday smoking to quitting smoking—were the effective intervention strategies.

The accuracy and AUROC of DQN models are shown in Multimedia Appendix 3. We used the decrease of lung cancer incidence to evaluate the effectiveness of the transfer strategy. The transfer strategy brought the largest decline of lung cancer incidence compared to others, which could be considered an optimal transfer strategy. Otherwise, the transfer strategy should be adjusted through a feedback mechanism. The whole workflow would be reworked as shown in Figure 4; the transfer pathway was comprehensively evaluated until an optimal transfer strategy was derived. For off-policy evaluation, DQN used experience replay data to update the strategy, and existing historical data were reused, as shown in Figure 4. Extending the duration of smoking cessation and reducing the frequency of smoking were the optimal transfer strategies for risk improvement pathway. The tangible intervention strategies were extending the duration of smoking cessation from less than 1 month to more than 5 years and reducing smoking frequency from everyday smoking to quitting. A forest plot of lung cancer incidence and smoking is shown in Figure 9. Lung cancer incidence of the high risk group had the closest relationship with smoking than the medium risk, low risk, and non-risk groups. The optimal transfer pathways and the decline of lung cancer incidence of stratified risk groups are displayed in Table 3. Lung cancer incidence of the high risk group had declined from 1060.38 per 100,000 to 336.78 per 100,000, which had decreased by 68.2% through risk transfer. The lung cancer incidence of the medium risk group had reduced from 671.20 per 100,000 to 289.13 per 100,000, which had declined by 56.9%.

The incidence trend of lung cancer for stratified risk groups was simulated and analyzed to evaluate the optimal transfer pathway and transfer strategy, as shown in Figure 10. According to the optimal transfer strategy and the optimal transfer pathway, lung cancer incidence of the high risk group reduced to 106.78 per 100,000 after simulative intervention and risk transition, lung cancer incidence of the medium risk group declined to 29.13 per 100,000, and the incidence of the low risk group also decreased obviously. The results validated the effectiveness and availability of the transfer pathway.

We used random forest to conduct comparison work of the DQN-based risk transfer pathway simulation. The training dataset and internal test dataset of HRS were leveraged to construct a random forest model for lung cancer incidence, lung cancer survival, and overall survival. The CHARLS dataset was used for external validation. Risk transfer pathways of lung cancer incidence were derived through random forest models, and optimal transfer pathways were also deduced, which were mainly the same as those of the DQN models. For example, in the risk improvement pathway, extending the duration of smoking cessation and reducing the frequency of smoking were the optimal transfer strategies through random forest models, which were consistent with the DQN models. The accuracies of random forest external validation models for stratified risk groups were from 0.823 to 0.851, and the AUROCs were from 0.778 to 0.836, which were almost lower by 10% than the DQN models, as shown in Table 4. Therefore, DQN models had better performance for transfer pathways computation than random forest models.