In the United States, asthma affects 8.4% of the population and leads to 2.1 million emergency department (ED) visits, 479,300 hospitalizations, 3388 deaths, and US $50.3 billion in cost annually [1,2]. Reducing hospital encounters, including inpatient stays and ED visits, is highly desired for asthmatic patients. For this purpose, using prognostic predictive models to prospectively identify high-risk asthmatic patients and enroll them in care management for tailored preventive care is deemed state of the art and has been adopted by health plans in 9 of 12 metropolitan communities [3]. Once enrolled, care managers make regular phone calls to help patients book appointments and schedule health and related services. If done properly, this can cut the patients’ future hospital encounters by up to 40% [4-7].

Unfortunately, the current high-risk patient identification methods have major gaps, leading to suboptimal outcomes. Care management typically enrolls only 1% to 3% of patients because of capacity constraints [8]. The existing models for predicting hospital encounters in asthmatic patients are inaccurate, which is reflected by their area under the receiver operating characteristic curve (AUC) ≤0.81 [9-22]. When used for care management, these models miss over half of the patients who will incur future hospital encounters and incorrectly classify many other patients as patients who will incur future hospital encounters. This makes it difficult to align care management enrollment with the patients who will actually incur future hospital encounters, increasing health care costs and impairing patient outcomes. If we could find 5% more asthmatic patients who would incur future hospital encounters and enroll them in care management, we could improve outcomes and avoid up to 9850 inpatient stays and 36,000 ED visits each year [1,4-7].

The goal of this study was to develop a more accurate model for predicting hospital encounters for asthma in asthmatic patients. The dependent variable is categorical with 2 possible values: whether future hospital encounter for asthma will occur or not. Accordingly, our model employs clinical and administrative data to perform binary classification, with the intention to better guide care management allocation and improve outcomes for asthmatic patients. A description of the development and evaluation of our model follows.

In this study, we conducted secondary analysis of retrospective data. The study was reviewed and approved by the institutional review boards of Intermountain Healthcare, University of Utah, and University of Washington Medicine.

Our patient cohort was based on the patients who visited Intermountain Healthcare facilities during 2005 to 2018. Intermountain Healthcare is the largest health care system in the Intermountain region (Utah and southeastern Idaho), with 185 clinics and 22 hospitals providing care for approximately 60% of the residents in that region. The patient cohort included asthmatic patients identified as residents of Utah or Idaho, with or without a specific home address. A patient was defined as having asthma in a given year if the patient had at least one diagnosis code of asthma (International Classification of Diseases, Ninth Revision [ICD-9]: 493.0x, 493.1x, 493.8x, and 493.9x; International Classification of Diseases, Tenth Revision [ICD-10]: J45.x) in that year in the encounter billing database [11,23,24]. Patients who died during that year were excluded. There were no other exclusions.

In the rest of this paper, we use hospital encounter for asthma to refer to inpatient stay or ED visit at Intermountain Healthcare with a principal diagnosis of asthma (ICD-9: 493.0x, 493.1x, 493.8x, and 493.9x; ICD-10: J45.x). For each patient meeting criteria for asthma in a given year, we looked at any hospital encounter for asthma in the following year as outcome. In our modeling, we used each asthmatic patient’s data by the end of each year to predict the patient’s outcome in the following year.

The Intermountain Healthcare enterprise data warehouse provided a structured, clinical, and administrative dataset, including all visits of the patient cohort at Intermountain Healthcare facilities during 2005 to 2018.

Following the approach outlined in our study design papers [25,26], we considered 235 candidate features derived from the structured attributes in our dataset. These features came from 4 sources: the >100 potential risk factors for asthma exacerbations reported in the literature [9,22,27-34]; features used in the existing models for predicting asthma exacerbations [9-22]; factors impacting patients’ general health status mentioned in the literature [31,35,36]; and features suggested by the clinical experts in our team—MDJ, BLS, and FLN. As the characteristics of the patient, the care provider, and the treating facility impact the patient’s outcome, we used patient features as well as provider and facility features [25,26].

The 235 candidate features are listed in the first table in Multimedia Appendix 1 [37-39], where each reference to the number of a specific type of items, such as medications, counts multiplicity, unless the word distinct appears. A major visit for asthma is defined as an outpatient visit with a primary diagnosis of asthma, an ED visit with an asthma diagnosis code, or an inpatient stay with an asthma diagnosis code. An outpatient visit with asthma as a secondary diagnosis is defined as a minor visit for asthma. Intuitively, all else being equal and compared with a patient with only minor visits for asthma, a patient with 1 or more major visits for asthma is more likely to incur future hospital encounters for asthma.

Each input data instance for the predictive model includes the 235 candidate features, targets the unique combination of an asthmatic patient and a year (index year), and is used to predict the patient’s outcome in the following year. For that combination of patient and year, the patient’s age, current primary care provider (PCP), and home address were determined based on the data available on the last day of the index year. The features of premature birth, bronchiolitis, duration of asthma, duration of chronic obstructive pulmonary disease, whether the patient had any drug or material allergy, whether the patient had any environmental allergy, whether the patient had any food allergy, and the number of allergies of the patient were derived from the historical data from 2005 to the index year. Furthermore, 1 feature was derived from the historical data in both the index year and the year before. This feature is as follows: the proportion who incurred hospital encounters for asthma in the index year out of all asthmatic patients of the patient’s current PCP in the year before. The remaining 226 features were derived from the historical data in the index year.

Recall that each data instance targets a unique combination of an asthmatic patient and a year. Tables 2 and 3 exhibit the demographic characteristics of our patient cohort during 2005 to 2016 and 2017, respectively. The characteristics are relatively similar between the 2 periods. During 2005 to 2016 and 2017, about 3.59% (11,332/315,308) and 4.22% (812/19,256) of data instances linked to hospital encounters for asthma in the following year, respectively.

On the basis of chi-square 2-sample test, for both 2005 to 2016 and 2017 data, the data instances linked to future hospital encounters for asthma and those linked to no future hospital encounter for asthma showed the same distribution for long-acting beta2-agonist prescription (P=.67 for the 2005 to 2016 data and P=.11 for the 2017 data), mast cell stabilizer prescription (P=.29 for the 2005 to 2016 data and P>.99 for the 2017 data), allergic rhinitis occurrence (P=.38 for the 2005 to 2016 data and P=.13 for the 2017 data), and cystic fibrosis occurrence (P=.21 for the 2005 to 2016 data and P=.20 for the 2017 data) and, they showed different distributions for gender (P<.001 for the 2005 to 2016 data and P=.002 for the 2017 data), race (P<.001), ethnicity (P<.001), insurance category (P<.001), inhaled corticosteroid prescription (P<.001), inhaled steroid and rapid-onset long-acting beta2-agonist combination prescription (P<.001 for the 2005 to 2016 data and P=.002 for the 2017 data), leukotriene modifier prescription (P<.001), inhaled short-acting beta2-agonist prescription (P<.001), systemic corticosteroid prescription (P<.001), anxiety or depression occurrence (P<.001 for the 2005 to 2016 data and P=.002 for the 2017 data), bronchopulmonary dysplasia occurrence (P<.001 for the 2005 to 2016 data and P=.02 for the 2017 data), chronic obstructive pulmonary disease occurrence (P<.001), eczema occurrence (P<.001), gastroesophageal reflux occurrence (P<.001), obesity occurrence (P<.001 for the 2005 to 2016 data and P=.004 for the 2017 data), premature birth occurrence (P<.001), sleep apnea occurrence (P<.001), and smoking status (P<.001). For the data from 2005 to 2016, different distributions were shown for sinusitis occurrence (P=.006). For the 2017 data, the same distribution was shown for sinusitis occurrence (P=.91). On the basis of the Cochran-Armitage trend test [47], for both 2005 to 2016 and 2017 data, the data instances linked to future hospital encounters for asthma and those linked to no future hospital encounter for asthma showed different distributions for age (P<.001) and duration of asthma (P<.001).

After finishing the search process to maximize the AUC, our automatic model selection method [45] chose the XGBoost classification algorithm [43] and the hyperparameter values listed in Multimedia Appendix 1. XGBoost is based on decision trees and can deal with missing feature values naturally. As XGBoost only accepts numerical features as its inputs, each categorical feature was first converted into 1 or more binary features via one-hot encoding before being given to XGBoost. Our final model was constructed using XGBoost and the 142 features listed in the descending order of their importance values in the second table in Multimedia Appendix 1. Due to having no extra predictive power, the other features were automatically removed by XGBoost. As detailed in the book by Hastie et al [48], XGBoost automatically computed each feature’s importance value as the mean of such values across all decision trees in the XGBoost model. In each tree, the feature’s importance value was computed based on the performance improvement gained by the split at each internal node of the tree using the feature as the splitting variable, weighted by the number of data instances the node is responsible for.

Our final model reached an AUC of 0.859 (95% CI 0.846-0.871). Figure 1 shows our final model’s receiver operating characteristic curve. Table 4 shows our final model’s performance metrics when differing top percentages of asthmatic patients with the highest predicted risk were used as the cutoff threshold for conducting binary classifications. When this threshold was at 10.00% (1926/19,256), our final model reached an accuracy of 90.31% (17,391/19,256; 95% CI 89.86-90.70), a sensitivity of 53.7% (436/812; 95% CI 50.12-57.18), a specificity of 91.93% (16,955/18,444; 95% CI 91.54-92.31), a PPV of 22.65% (436/1925; 95% CI 20.74-24.61), and an NPV of 97.83% (16,955/17,331; 95% CI 97.60-98.04). Table 5 shows the corresponding confusion matrix of our final model.

Recall that several features require more than 1 year of historical data to compute. If we exclude these features and use only those features computed on 1 year of historical data, the model’s AUC degrades to 0.849.

Without excluding the features that require more than 1 year of historical data to compute, the model trained on both asthmatic adults’ (age ≥18 years) and asthmatic children’s (age <18 years) data reached an AUC of 0.856 on asthmatic adults and an AUC of 0.830 on asthmatic children. In comparison, the model trained only on asthmatic adults’ data reached an AUC of 0.855 on asthmatic adults. The model trained only on asthmatic children’s data reached an AUC of 0.821 on asthmatic children.

If we used only the top 21 features listed in the second table in Multimedia Appendix 1 with an importance value ≥0.01 and excluded the other 121 features, the model’s AUC degraded from 0.859 to 0.855 (95% CI 0.842-0.867). When the cutoff threshold for conducting binary classification was set at the top 10.00% (1926/19,256) of asthmatic patients with the highest predicted risk, the model’s accuracy degraded from 90.31% (17,391/19,256) to 90.14% (17,357/19,256; 95% CI 89.74-90.58), sensitivity degraded from 53.7% (436/812) to 51.6% (419/812; 95% CI 48.02-55.24), specificity degraded from 91.93% (16,955/18,444) to 91.83% (16,938/18,444; 95% CI 91.43-92.24), PPV degraded from 22.65% (436/1925) to 21.77% (419/1925; 95% CI 20.03-23.68), and NPV degraded from 97.83% (16,955/17,331) to 97.73% (16,938/17,331; 95% CI 97.49-97.95).

We built a more accurate machine learning classification model to predict hospital encounters for asthma in the following year in asthmatic patients. Our final model achieved a higher AUC than what has been reported in the literature for this task [9-22]. After further refinement to improve its accuracy and to automatically explain its prediction results [49,50], our final model could be integrated into an electronic medical record system to guide care management allocation for asthmatic patients. This could better allocate a scarce and expensive resource and help improve asthma outcomes.

Asthma in adults is different from asthma in children. Our final model reached a higher AUC on asthmatic adults than on asthmatic children. More work is needed to understand the reason for this difference. In addition, more work is needed to improve the prediction accuracy on asthmatic children compared with asthmatic adults.

We considered 235 features in total, about 60% of which appeared in our final model. If a feature is unused by our final model, it does not necessarily mean that this feature has no predictive power. Rather, it only shows that this feature offers no extra predictive power on our specific dataset beyond what the features used in our final model have. On a larger dataset with more asthmatic patients, it is possible that some of the excluded features will provide extra predictive power. This is particularly true with features whose nontrivial values occur on only a small portion of asthmatic patients, such as a comorbidity with a low prevalence rate. When too few data instances take nontrivial values, the features’ predictive power may not appear.

In the second table in Multimedia Appendix 1, the 2 most important features, as well as several within the top 20, reflect overall instability of the patient’s asthma. The instability could derive from physiologic characteristics of the patient’s asthma, as reflected by the maximum blood eosinophil count, the maximum percentage of blood eosinophils, and the average respiratory rate. The instability could also result from treatment noncompliance, PCP changes, insurance changes, and socioeconomic issues for which data were unavailable.

Researchers have developed multiple models to predict inpatient stays and ED visits in asthmatic patients [9-22]. Table 6 compares our final model with these models, which include all relevant ones mentioned in Loymans et al’s recent systematic review [9]. None of these models obtained an AUC >0.81, whereas our final model’s AUC is 0.859. In other words, compared with our final model, each of these models reached an AUC lower by at least 0.049. Compared with prior model building, our model building assessed more candidate features with predictive power, adopted a more advanced classification algorithm, and used data from more asthmatic patients. All of these helped boost our final model’s accuracy. Our principle of considering extensive candidate features to help enhance the model’s accuracy is general and can be applied to other diseases and outcomes such as health care cost [51].

Except for Yurk et al’s model [17], all other prior models had a PPV ≤22% and a sensitivity ≤49%, which are lower than those achieved by our final model. Yurk et al’s model [17] obtained better sensitivity and PPV primarily because the model used a different prediction target: hospital encounters or ≥1 day lost because of reduced activities or missed work for asthma. This prediction target occurs for more than half of the asthmatic patients, making it relatively easy to predict. If the prediction target were changed to hospital encounters for asthma, a rarer outcome that is harder to predict, we would expect the sensitivity and PPV reached by Yurk et al’s model [17] to drop.

Despite being more accurate than the prior ones, our final model still reached a relatively low PPV of 22.65% (436/1925). However, this does not prevent our final model from being clinically useful because of the following reasons:

Our final model used 142 features. Reducing features used in the model could ease its clinical deployment. For this, one could use the top few features with the highest importance values (eg, ≥0.01) and exclude the others, if one is willing to accept a not-too-big degrade of model accuracy. Ideally, one should first assess the features’ importance values on a dataset from the target health care system before deciding which features should be kept for that system. A feature’s importance value varies across different health care systems. A feature with a low importance value on the Intermountain Healthcare dataset might have a decent importance value on a dataset from another health care system. Similar to the case with many other complex machine learning models, an XGBoost model using a nontrivial number of features is difficult to interpret globally. As an interesting area for future work, we are in the process of investigating using the automatic explanation approach described in our prior papers [49,50] to automatically explain our final XGBoost model’s prediction results on individual asthmatic patients.

Our final model was built using the XGBoost classification algorithm [43]. For binary classification with 2 unbalanced classes, XGBoost uses a hyperparameter scale_pos_weight to control the balance of the weights for the positive and negative classes [54]. One could set scale_pos_weight to the ratio of the number of negative data instances to the number of positive data instances [54], although the optimal value of scale_pos_weight often deviates from this value by a degree varying by the specific dataset. In our case, to maximize the model’s AUC, our automatic model selection method [45] did a search of possible hyperparameter values and eventually set scale_pos_weight to a nondefault value to balance the 2 classes of future hospital encounters for asthma or not [55]. This has the side effect of making the model’s predicted probabilities of incurring future hospital encounters for asthma very small and unaligned with the actual probabilities [55]. This side effect does not prevent us from selecting the top few percentage of asthmatic patients with the highest predicted risk as candidates for receiving care management or other preventive interventions. To avoid this side effect, we could set scale_pos_weight to its default value of 1, without balancing the 2 classes. However, that would degrade the model’s AUC from 0.859 to 0.849 (95% CI 0.836-0.862).

This study has several limitations, all of which provide interesting areas for future work:

Our final model improves the state of the art for predicting hospital encounters for asthma in asthmatic patients. In particular, our final model reached an AUC of 0.859, which is higher than those previously reported in the literature for this task by ≥0.049. After further refinement, our final model could be integrated into an electronic medical record system to guide allocation of scarce care management resources for asthmatic patients. This could help improve the value equation for asthma care by improving asthma outcomes while also decreasing resource use and cost.