Using Machine Learning to Predict the Duration of Atrial Fibrillation: Model Development and Validation

Introduction

Atrial fibrillation (AF) is a progressive condition, with many cases starting as paroxysmal AF (PAF) and progressing to persistent AF (PeAF) and long-standing persistent AF (LsPeAF). PeAF is classified as AF lasting for less than 1 year, while LsPeAF is classified as AF lasting for more than 1 year [1,2]. Catheter ablation (CA) is a beneficial treatment for patients with AF, but the outcomes for LsPeAF are not as good as those for PAF or PeAF, as atrial degeneration becomes advanced due to long-term AF persistence [3]. Therefore, in the 2017 Heart Rhythm Society/European Heart Rhythm Association/European Cardiac Arrhythmia Society statement, the recommendation for CA for symptomatic LsPeAF is lower than that for PAF and PeAF [4]. The 2023 American College of Cardiology/American Heart Association/American College of Clinical Pharmacy/Heart Rhythm Society guidelines also state that CA is particularly effective in patients with AF of less than a 1-year duration [5], and the 2020 European Society of Cardiology guidelines list a longer duration of AF as a risk factor for recurrence when considering the indication for CA [6]. In addition, pulmonary vein isolation alone is often insufficient in cases with advanced atrial degeneration, as the involvement of maintenance mechanisms as well as triggering of AF is greater. In cases with LsPeAF, additional treatments other than pulmonary vein isolation such as linear ablation [7,8], complex fractionated atrial electrogram ablation [9], driver ablation [10,11], and low voltage area ablation [12] may also be considered. Estimation of the duration is therefore an important factor in determining the treatment strategy. Although the AF duration is an important finding for clinicians when deciding on a treatment strategy for AF, we often experience patients whose AF duration is unknown from the patient interview. In such cases, the cardiologists judge the AF duration based on clinical findings, such as an advanced age, a history of hypertension or heart failure, and the presence or absence of left atrial enlargement [13]. Thus, the objective of this study was to “improve the accuracy of the cardiologists’ diagnosis of the AF duration,” and the steps to achieve this objective were to “develop an AF duration prediction model” and to “validate the support performance of the prediction model.”

A small number of machine learning (ML) models have been developed to estimate the AF duration using only the characteristics of the fibrillation wave (f wave) on the 12-lead electrocardiogram (ECG) during AF. For example, the area under the curve (AUC) for a prediction using the amplitude of lead II was 0.77 and the AUC for a prediction using the dominant frequency (DF) in lead V1 was 0.63 [14]; a study reporting the prediction of the AF duration with f wave training using an ML model also had an AUC of 0.78 [15]. Therefore, in this study, we trained the ML prediction model based on multimodal data, including the ECG f wave characteristics, patient background, echocardiographic findings, and laboratory data.

Methods

Study Design and Population

The study included 272 patients with PeAF (aged 20-90 years) who were admitted for an initial CA at the Japanese Red Cross Kyoto Daini Hospital between January 1, 2015, and December 31, 2023, with available ECGs and patient background, echocardiography, and laboratory data within 3 months. A total of 189 (69.5%) patients were included in the study, excluding 14 (5.1%) patients whose ECG recorded a pacemaker rhythm, 30 (11%) whose QRS waves were difficult to exclude or whose extracted f waves were too short for a feature analysis, and 39 (14.3%) whose AF duration was difficult to determine.

The following methods were used to define PeAF (AF duration of less than 1 year) and LsPeAF (AF duration of 1 year or more). First, PeAF was defined if any of the following conditions were met: (1) sinus rhythm was recorded on the ECG within 1 year, excluding postdefibrillation ECGs, followed by a consecutive recording of AF; and (2) the onset of clinical symptoms or an irregular pulse clearly related to AF was confirmed within 1 year, followed by a consecutive ECG recording of AF. Second, LsPeAF was defined as any of the following: (1) patients with ECG recordings of AF confirmed for more than 1 year consecutively and (2) patients with onset of clinical symptoms or an irregular pulse clearly related to AF more than 1 year prior, followed by consecutive ECG recordings of AF within 1 year. Cases that did not meet any of the above conditions were excluded as cases with an unknown duration.

The ML model was developed and cross-validated to determine whether the duration of AF was more than 1 year, using 145 patients with a registration period from January 1, 2015, to March 31, 2023, as training data. The predictive ability of the ML model was tested using 44 patients whose registration period was from April 1, 2023, to December 31, 2023, as test data (Figure 1).

In phase 1, a total of 10 cardiologists (group A) evaluated the test data of 44 patients using their ECGs and patient information and answered whether each patient had an AF of at least a 1-year duration. In phase 2, they answered that again after being presented with the predictive results of the ML model and the Shapley additive explanation (SHAP) values, which calculated the contribution of each variable feature to the predictive results of the model. The case presentation in phases 1 and 2 is shown in Figure 2. Following that, group B was shown with the diagnostic results of group A, and the same 2-phase tests were conducted as in group A, with a prior recognition of the cardiologists’ ability to diagnose the AF duration. Finally, all cardiologists were asked whether they attached any importance to each of the characteristics as a decision-making factor.

‎

‎

Data Analysis

Feature Analysis From the 12-Lead ECG Data

The heart rate (HR), QRS width, S1 amplitude, RV5 amplitude, and R+S (=S1 amplitude+RV5 amplitude) were referenced from 10-second ECG data analyzed using a 12-lead ECG recording device (FCP-8800 or FCP-8700; Fukuda Denshi Co, Ltd).

F Wave Extraction

The f wave was extracted from 10-second ECG data. Bandpass filtering was used to reduce any baseline meandering and high-frequency noise. The f waves were in the 4-9 Hz bandwidth, so the cutoff frequency was set to 0.8-40 Hz. The QRST interval was detected for QRST cancellation; the R wave time was peak-detected by applying the Pan-Tompkins algorithm. The Q wave time was obtained by subtracting 37 ms, the typical ventricular activation time, from the R wave time; the T wave time was obtained by adding 200 ms to the R wave time. For each QRST interval, the f wave was extracted using a principal component analysis. To suppress any residual QRST features, filtering was performed by a bandpass filter. The f waves were in the 4-9 Hz bandwidth, so the cutoff frequency was set to 3-20 Hz (Figure 3).

‎

F Wave Features

The root mean square (RMS) of the amplitude was defined as the square root of the mean square, which is the arithmetic mean of the squares of the signal amplitudes in the time domain. Sample entropy (SampEn) was used in the evaluation of f wave irregularities. The SampEn is a measure of entropy first proposed by Richman and Moorman [16], a method designed to reduce the bias of approximate entropy and to obtain more robust statistics. The SampEn was defined by the following equation:

where is (N – m)^–1 times the number of that meets and is (N – m)^–1 times the number of that meets for all 1 ≤ j ≤ N – m.

A threshold value of 3.5 and a sample size of 3 were set. After a fast Fourier transform, the DF and organization index (OI) were determined. The DF is one of the most widely used indices for the frequency analysis of f waves, and the frequency with the highest power value in the frequency distribution is defined as the DF. The OI was used as a measure of f wave organization. The OI was defined as the ratio of the area under the highest peak and its harmonics (not including the fifth harmonic peak) to the rest of the spectrum in the 3-15 Hz bandwidth.

Statistical Analysis

Continuous variables are presented as the mean (SD). Categorical variables are presented as frequencies (percentages). Statistical significance was determined using the Mann-Whitney U test for continuous variables and the chi-square test for categorical variables; a P value <.05 was considered statistically significant. Python software (version 3.10.12; Python Software Foundation) was used for statistical analysis.

The percentage of correct answers for each cardiologist was calculated from the results of the cardiologist’s test and compared between phase 1 and phase 2 in group A and group B, respectively, using a paired sample 2-tailed t test.

ML Model and Cross-Validation

The model to predict long-term PeAF was built based on an ML algorithm. The objective variables were PeAF (duration of less than 1 year) and LsPeAF (duration of more than 1 year) defined earlier. The model construction was performed in the following steps: (1) step 1 involves the investigation of the ML algorithms; (2) step 2 involves feature selection; and (3) step 3 includes model training and evaluation.

Step 1

To select the appropriate ML algorithm for the research data, we used DataRobot to investigate more than 1000 different types of ML algorithms. The results of the survey showed that the gradient-boosting decision tree was highly accurate for the study dataset. Therefore, 2 ML algorithms, XGBoost (extreme gradient boosting) and LightGBM (a gradient-boosting decision tree model), were selected for the model development.

Step 2

In order to prevent overfitting with the ML algorithms, an appropriate number of feature selections was performed. To account for the effect of multicollinearity on the model, one of the features was excluded from the features with a correlation coefficient between the features greater than 0.7. The feature selection used a stepwise variable reduction method to search for the optimal combination of features. In particular, the Boruta package [17] was used to exclude any features that did not contribute to discriminating between PeAF and LsPeAF.

Four models were developed according to the features in order to evaluate the incremental benefits gained by adding each feature (Multimedia Appendix 1). Model 1 consisted of the age, sex, body size (height, weight, BMI, and body surface area), and past history. Model 2 further included the echocardiographic data (left atrial diameter [LAD], ejection fraction, and mitral regurgitation greater than moderate). Model 3 also included the ECG data (HR, QRS width, SV1 amplitude, RV5 amplitude, and R+S). Model 4 also included the f wave features (RMS, DF, entropy, and OI). Those modalities were designed according to common clinical measures.

Step 3

The dataset was divided into training data (145/189, 76.7%) and test data (44/189, 23.3%). The test data were set to be in the future in time relative to the training data. No test data were set for external validation. The training data were divided into 5 partitions and stratified sampling was performed to ensure an equal target distribution. The ML models were trained using Bayesian optimization methods with a nested 5-partition cross-validation.

For the prediction models selected from the aforementioned evaluations, the TreeExplainer of the SHAP method was used to estimate the extent to which each feature contributed to the decision-making of the prediction model. The feature importance was arranged in descending order based on SHAP values.

The cutoff points for calculating the other evaluation metrics—accuracy, F₁-score, sensitivity, specificity, positive predictive value, and negative predictive value—were Youden indices computed from the training dataset.

Ethical Considerations

The study was approved by the Medical Ethics Review Committee of the Kyoto Prefectural University of Medicine (ERB-C-2195-1). Informed consent was given on an opt-out basis, and all data were anonymized. The study was conducted without offering any financial or material rewards to the participants.

Results

Clinical Characteristics

Of the 189 patients studied, 130 (68.8%) had PeAF and 59 (31.2%) had LsPeAF (Table 1). There was no significant difference in age (P=.50), and in terms of sex, there were significantly more male patients in the LsPeAF group (P=.01). There were no significant differences in the medical history between the 2 groups (all P>.05). On examination, echocardiography showed no significant difference in LAD between the two groups (P=.11). The ECG data showed that the HR was significantly lower in the LsPeAF group (P<.001), but there were no significant differences in the QRS wave width or amplitude (all P>.05).

Among the f wave characteristics, the aVR and V1 RMS were significantly lower in the LsPeAF group (P=.05 and P=.04, respectively), and the other inductions tended to have lower mean values overall, but there were no significant differences (all P>.05). No significant differences were observed in the DF and OI in any of the inductions (all P>.05; Table 2).

Predictive Accuracy of the ML Models

In this study, model 4, trained with XGBoost, which had the highest AUC among the ML models created, was adopted (Multimedia Appendices 2 and 3). To evaluate the prediction accuracy of the ML model, a random seed was set 5 times, and the mean receiver operating characteristic–AUC curve was calculated; the mean AUC of those 5 was 0.82 (Figure 4).

‎

In the SHAP value, the ML model gave the highest importance to HR. The following weightings were then assigned in the following order: RMS, SampEn, amplitude, left ventricular ejection fraction, sex, OI, BMI, QRS wave amplitude, QRS width, LAD, and age (Figure 5).

The predicted results for the test data were 0.82 for accuracy, 0.72 for sensitivity, 0.81 for specificity, 0.81 for positive predictive value, 0.82 for negative predictive value, and 0.77 for F₁-score.

‎

Cardiologists’ Decisions With and Without ML Model Support

The diagnostic accuracy of the AF duration by cardiologists is shown in Figure 6. The mean percentage of correct answers in group A was 63.9% (SD 9.6%) for phase 1 and improved to 71.6% (SD 9.3%) for phase 2 (P=.01). The mean percentage of correct responses in group B was 59.8% (SD 5.3%) for phase 1 and improved to 68.2% (SD 5.9%) for phase 2 (P=.007). There was no difference in the percentage of correct answers between groups A and B for phase 2 (P=.48). The diagnostic accuracy with the cardiologists significantly improved with ML model support in both groups. The mean percentage of answers that differed from the predictions of the ML model for phase 2 (percentage of answers where cardiologists did not trust the ML model and believed their own determination) was 17.3% (SD 10.3%) in group A and 20.9% (SD 5%) in group B, and there was no significant difference between groups A and B (P=.85).

‎

Posttest Questionnaire for Cardiologists

When a posttest questionnaire was conducted to find out which characteristics were important to the 20 cardiologists, 18 (90%) said that the LAD was important. This was followed by 17 (85%) cardiologists selecting the f wave amplitude in the 12-lead ECG, 16 (80%) selecting age, and 15 (75%) selecting the HR in the 12-lead ECG (Table 3).

Discussion

Principal Findings

Regarding the prediction of AF duration, the goal was to achieve a more accurate prediction by using ML models. As a result, the part that had previously relied on the cardiologists’ experience and subjectivity was supplemented, and the uncertainty related to the prediction was reduced. Next, we adopted an approach that combined the cardiologists’ diagnostic capability with the ML models. This enabled the objective prediction with the ML model to be referenced against the cardiologists’ experience and subjective judgment, thereby improving the overall quality of the cardiologists’ diagnostic process.

Performance of the ML Model and the Cardiologists’ Reaction

There have been few reports that have validated the prediction of the duration using f wave characteristics: the AUC for the prediction using the amplitude of the f wave in lead II of the 12-lead ECG was 0.77, the AUC for the prediction using the DF of the f wave in lead V1 was 0.63 [14], and the AUC for the prediction using f wave learning with the ML model was 0.78 [15]. The ML model developed in this study was trained not only with the ECG characteristics including the f wave but also a past history and echocardiographic and laboratory data, which could improve the prediction accuracy to AUC 0.82 despite the small dataset. A validation using the test data also confirmed the result of an accuracy of 0.818. Therefore, in both groups A and B, the support of the ML model improved the ability of the cardiologists to diagnose the AF duration.

On the other hand, group A had a different answer selected for 24.3% of the ECGs compared to the ML model. One possible reason for that was likely to be that they believed their own decision over the ML support, as they did not know the superiority or inferiority of the cardiologists and ML model in predicting the duration of AF.

Therefore, an additional test was conducted with group B. The accuracy of the cardiologists’ diagnosis (percentage of correct answers in group A) was shown to group B in advance so that they were aware of the superiority or inferiority between the accuracy of the cardiologists’ diagnosis and the accuracy of the ML model. The same diagnostic test was then conducted with group B as in group A. Surprisingly, the percentage of cases selected that differed from the prediction of the ML model (21%) did not improve. That may have been due to the feature set of the model.

Among the SHAP values, the HR was the most correlated with the AF duration. Those facts are empirically recognized by cardiologists. In fact, 75% (15/20) of cardiologists consider HR to be important. For example, animal studies have shown that significant functional electrophysiological remodeling of the atrioventricular node occurs after prolonged AF, resulting in a spontaneous decrease in the ventricular HR [18]. Aging has also been shown to correlate with the atrioventricular node dysfunction as a result of atrioventricular node remodeling and ion channel expression [19]. In addition, patients with LsPeAF are often treated with drugs that control the HR, such as beta-blockers, calcium channel blockers, and digitalis. Those drugs decrease the HR and prevent it from becoming excessively fast. Taking those drugs over a long period of time may decrease the HR due to their cumulative effect. The dataset used in this study consisted of patients with AF scheduled for CA, and patients with a particularly large LAD tended to be excluded from the dataset at the discretion of their cardiologists because of their high risk of recurrence after CA. That selection bias may have been responsible for the discrepancy between the LAD characteristics and the ML prediction and cardiologist decision-making. The LAD can also be magnified by factors other than left atrial remodeling due to AF; Tsang et al [20] reported that left atrial enlargement is associated with congestive heart failure, vascular disease, transient ischemic attacks or strokes, and a history of smoking and is particularly strongly associated with diastolic dysfunction. Even in patients without a history of atrial arrhythmias or valvular disease, left atrial enlargement has been shown to increase with the severity of diastolic dysfunction, and remodeling due to AF (ie, LsPeAF) is not necessarily correlated with left atrial enlargement.

Challenges for the Clinical Application of ML Models for Clinical Assistance

The lack of a difference in the diagnostic improvement between groups A and B indicated that cardiologists tend to trust their own experience when there is a discrepancy between their perception and the results of the ML model. That may be the case unless, for example, their judgment was clearly erroneous, in which case simply recognizing that the accuracy of the ML model was higher than the cardiologist’s diagnostic accuracy may not have been sufficient to influence their decision-making. Lee [21] noted that artificial intelligence (AI) and human predictive performance may differ depending on the type of patient, and a deeper understanding of AI and human predictive capabilities, respectively, will increase human reliance on AI. Therefore, for clinicians to trust AI, understanding why the algorithm came to the conclusion it did is important for them to trust AI. In this regard, in addition to the diagnostic explainability, one should also consider expressing predictions as probabilities. That is because greater clarity about the uncertainty of predictions may help cardiologists assess the risk and reliability of a diagnosis. For example, according to a study by Heuer and Breiter [22], users of ML models value not only the accuracy of the prediction but also the probability representation of the prediction. In other words, they may need a process that allows them to decide whether to perform additional tests or treatments based on the reasons for the prediction (visual presentation of the impact of each factor) and its probability.

Limitations

This study had some limitations. First, the quality of the echo and ECG data and the problem of missing data may have affected the prediction accuracy of the ML model. Therefore, data cleaning and filtering were performed to exclude low-quality data and proceed with the analysis. As a result, the model’s prediction accuracy was 81.8%, with a sensitivity of 72% and specificity of 89%, thus confirming a certain degree of robustness. Second, this was a retrospective study and there may be some patients for whom the determination of the AF duration was uncertain. The determination of whether the duration was more than 1 year was based not only on interviews with the patients but also on physical examinations by cardiologists and ECG recordings. Furthermore, unclear symptoms and cases in which the timing was difficult to determine were excluded from this study, so we believe that the reliability and accuracy of the duration determination were high. Third, the study was not validated at other sites and the generalizability of the results to other sites and different populations was not confirmed. Also, the relatively small size of the dataset and the possibility of overfitting may have limited the robustness and generalizability of the model. Therefore, overfitting was avoided by “removing as many irrelevant variables as possible,” “cross-validation,” “validation with multiple seed values,” and “validation with test data from a different time period than the training data.” Those results confirmed that, even with the current dataset, the ML model complemented the cardiologists’ diagnostic ability and provided effective assistance in predicting the duration. However, further validation of the robustness and generalizability of the model is needed through prospective and larger datasets. Fourth, predicting the clinical outcomes of CA requires the unification of the CA procedures, which can only be adequately validated in prospective studies. Due to the limitations of the retrospective data in this study, we determined that it was not appropriate to predict the clinical outcomes, but in the future, we will evaluate the predictions regarding clinical outcomes using prospective data with a uniform CA technique. Lastly, the effect of rate control drugs as a confounding factor may have affected the HR. However, in this study, we dared to use data reflecting the actual clinical environment to improve the applicability of the model. In clinical practice, rate control with drugs is common, and by training the data to reflect that situation, we built a more practical model.

Conclusions

An ML model was developed to diagnose the duration of AF. The diagnostic accuracy of the cardiologist was improved with the support of the ML model. However, cardiologists did not entirely rely on the prediction of the ML model, even if they were aware of the limitations of their diagnostic capability.