Stroke has a massive impact on patients, their caregivers, and the health system. Approximately 795,000 stroke cases occur annually in the United States. It is the fifth leading cause of death and the leading cause of long-term disability, with nearly 7 million stroke survivors in the United States. With an increasing aging population, it is estimated that nearly 4% of the US adult population would have had a stroke by 2030 [1]. It is all the more tragic that a substantial proportion of stroke survivors will be readmitted to the hospital: current 30-day all-cause readmission rates range from 6.5%‐24.3% and these rates increase to 30%‐62.2% within a year [2]. Mortality following the initial hospitalization is also substantial: the case fatality rate in the hospital is 5%‐7%, increasing to 13%‐15% at 30 days, and 25%‐30% by 1 year [3].

Nearly one-fifth of Medicare beneficiaries discharged from a hospital get readmitted within 30 days [4]. Spurred by a landmark paper by Jencks et al [5] hospitals across the United States have begun implementing strategies to identify and reduce avoidable readmissions. Furthermore, beginning in 2013, 30-day readmission rates for a hospital’s Medicare patients with acute myocardial infarction, heart failure, and pneumonia have been compared with the expected rate of readmissions, using risk adjustment to account for age, gender, medical diagnosis, and selected medical history. Under the Hospital Readmissions Reduction Program, Medicare penalizes hospitals for higher-than-expected rates of readmissions among patients with these diagnoses [6,7]. Additional conditions are likely to be added in the future.

Given the clinical and policy significance of stroke, identifying factors that influence readmission risk is important to assist clinicians and hospitals in the care of patients with stroke and to prevent the preventable. Many factors may contribute to a hospital’s readmission rate after stroke, including patient preferences, demographics, socioeconomic status, comorbidities, stroke severity, health system, clinical care or process, and health outcome. It is important that the risk adjustment mechanisms account for these factors so that hospitals caring for a more complex patient mix are not unduly penalized for readmissions.

The purpose of our investigation was to determine whether we could develop predictive models, specific to patients with stroke, for nonelective readmission or death within 30 days after hospital discharge. In these models, we assessed the ability of commonly used administrative data, newer predictors from the electronic health record (EHR), and 2 manually assigned stroke-specific scales (National Institutes of Health Stroke Scale [NIHSS] and modified Rankin scale [mRS]).

Between June 15, 2018, and April 29, 2020, there were 5808 hospitalizations for adult patients with ischemic stroke. Of these 5808 hospitalizations, 794 (13.7%) were excluded from the study analysis because the patient died during the index hospitalization (n=198, 3.4%), the hospitalization originated at a non-KPNC facility (n=504, 8.7%), or follow-up data was incomplete (n=92, 1.6%). The final study population included 5014 hospitalizations. The average age of the study population was 71.9 (SD 13.99) years and 49.4% (2477/5014) of patients were male. The racial and ethnic distribution was 52.1% (2614/5014) White, 15.6% (782/5014) Hispanic, 15% (751/5014) Asian, and 14.3% (718/5014) Black. The median initial NIHSS score was 4 (IQR 2-8). The average mRS score at discharge was 2.8 (SD 1.64). Mean length of stay was 6.5 days (SD 7.71), and 55% (2756/5014) were discharged home. Table 1 presents the other characteristics of the study population.

Out of 5014 hospitalizations, 688 (13.7%) had a nonelective readmission within 30 days or died within 30 days postdischarge (composite outcome). Out of 5014 hospitalizations, 492 (9.8%) had a readmission only within 30 days, 150 (3.0%) died with no readmission within 30 days, and 46 (0.9%) had both a readmission within 30 days and died within 30 days (see Figure 1 and Table 1). Patients who died but had no readmission within 30 days, had an elevated history of atrial fibrillation (47/150, 31.3%). Patients who had no readmission or death within 30 days tended to be more likely to have received alteplase during the index hospital stay (639/4326, 14.8%). The proportion of patients with full code orders at discharge varied across patient groups (range 8.7%-80.4%). The proportion of patients who were discharged home after the index event varied by outcome group. Of those who were both readmitted and died, 28% (13/46) had been released to home, compared with 40.7% (200/492) of those who were readmitted and alive, and 56.2% (2433/4326) of those who were alive and had no readmission. In total, 50% of patients (23/46) who died within 30 days but had no readmission within 30 days, had been released to a regular (noncustodial) skilled nursing facility. Furthermore, 4% (6/150) of patients with death but no readmission, were originally discharged to a custodial skilled nursing facility (Table 1).

For hospitalizations that resulted in readmission within 30 days or death within 30 days post-discharge, the patients tended to be older (median age 78, IQR 68-86 years vs 72, IQR 62-82 years), were less likely to be male (290/688, 42.2% vs 2187/4326, 50.6%; P<.001), and more likely to have a Charlson comorbidity score greater than or equal to 4 (387/688, 56.3% vs 1584/4326, 36.6%; P<.001). More patients without the composite outcome had full code care directive at discharge (3476/4326, 80.4% vs 402/688, 58.4%; P<.001; see Table 2).

Table 3 shows results from performance testing of several multivariable predictive models of the composite outcome. The best performing models were TSL (c-statistic=0.685) and TSL plus mRS score at discharge (c-statistic=0.694). The model with TSL+NIHSS performed similarly (c-statistic=0.684) to the model with TSL alone (Table 3).

We have completed a study where we developed an adult patient stroke-specific predictive model for a composite outcome of readmission within 30 days and mortality within 30 days postdischarge. Among the models that were tested, the TSL model performed the best. Adding manually assigned stroke scales such as initial NIHSS (as a measure of stroke severity) and mRS at discharge (as a measure of functional status) to clinical data from a comprehensive EHR did not improve the performance of the TSL model significantly. The previously published TSL model [20] included an extensive amount of data for each patient including the LAPS2, COPS2, discharge care directive, and previous hospitalizations in the 30 days preceding the index hospitalization. Some of these data (such diagnoses in the previous 12 months before hospitalization, vital signs, laboratory data, and code status) were unique to our dataset and were never examined in previously published studies on 30-day readmission or mortality poststroke.

Readmissions after stroke are associated with worse disability, higher mortality, and higher costs of care [26,27]. Reduction of 30-day readmission is a quality metric set by the Centers for Medicare & Medicaid Services. Publications on readmission have been more focused on reported global performance and other conditions, and less so on stroke. In addition, most previous studies excluded patients aged <65 years. Being able to predict readmission after stroke may help to identify high-risk patients for targeted interventions and postdischarge programs, limit preventable readmissions, and improve long-term outcomes.

Previously published studies on readmissions after stroke provided only limited data on patient-level factors in their risk analysis. A systematic review of stroke readmission in 2010 identified only 16 published studies that reported risk-adjustment models at the patient level [2]. Considerable variation in reported outcomes was identified: 30-day all-cause readmission rates ranged from 6.5% to 24.3%, 1-year all-cause readmission rates from 30% to 62.2%, 30-day stroke-related readmission from 7.4% to 9.4%, and 1-year stroke-related readmission from 10.5% to 31.1%. Patient characteristics that were found to be associated with stroke readmissions included age, longer index hospital length of stay, worse physical functioning after stroke, and increased number of hospitalizations before stroke. Although some of these included stroke severity measures, no consistency existed across the models. A statement from the American Heart Association on risk adjustment of ischemic stroke outcomes for comparing hospital performance emphasized a minimum list of patient-level variables including age, sex, stroke severity, comorbid conditions, and risk factors [3]. However, in order to improve the usability of risk-adjusted models, the statement recommended that researchers should try to identify other factors, especially those not typically measured in administrative databases. To our knowledge, no previously published models on 30-day readmission or 30-day mortality included the mRS at discharge (a functional status assessment).

A number of studies on readmission and mortality poststroke used Medicare database therefore restricting their cohorts by older age and limited the generalizability of their results [28,29]. Our 30-day readmission rate of 9.7% was lower than rates (12%‐14%) reported from those studies using Medicare data. However, advanced age is associated with increased readmission after ischemic stroke [29]. On the other hand, a recent analysis of 30-day readmission after ischemic stroke for adults 18 years and older using the 2019 Nationwide Readmission Database reported a rate of 9.7%, which was nearly identical to ours [30].

For our cohort, the rate for 30-day mortality postdischarge was 3%, and the inpatient mortality rate was 3.4%, totaling 6.4% for 30 days from admission. Studies from Canada and the United Kingdom reported a 30-day mortality rate from admission of about 12% [31,32]. A US study using Medicare data reported a 30-day mortality rate of 13.6% [33]. Our lower 30-day mortality rate may be a reflection of a cohort of patients with insurance and access to care, inclusion of younger patients (ages 18 years and older), and an excellent performance of acute stroke treatment time as previously published [9].

The current state of predictive analytics for hospital readmission has not evolved enough to guide effective prevention efforts [4,5,34,35]. In addition, medical record technology, along with other aspects of health care in the United States is both fragmented and—from an informatics perspective—heterogeneous. Some institutions, such as university medical centers and KPNC, have sophisticated EHRs, while others still use paper charts. Similarly, the ability of institutions to retrieve electronic data varies considerably, as does the timeliness of such retrieval. Given this heterogeneous information environment, it is important to (1) highlight desirable characteristics of future readmission predictive models for patients with stroke, and (2) develop models that can be instantiated in different settings (ie, ranging from those using only claims data to those that use comprehensive EHRs). This would permit the development of modular approaches that individual institutions could adapt to their specific circumstances.

It was not surprising that models using only administrative data (LACE) did not perform well. However, it was disappointing that the TSL model with inclusion of detailed EHR data (longitudinal comorbidity in the 12 months preceding index hospitalization, COPS2; and LAPS2) did not result in great discrimination. Adding the initial NIHSS score and the mRS score at discharge to the model did not improve its performance, perhaps because the TSL model at baseline already contained other data elements relating to patient status and severity of their condition while hospitalized. NIHSS and mRS were moderately correlated to LAPS2 score, LACE, and TSL. We did not have other potential measures of stroke severity such as infarct size from neuroimaging study, or change in mRS at discharge compared with baseline prehospital mRS, or mRS postdischarge.

Our study had several strengths. Our approach synthesizes two types of expertise: deep understanding of the clinical aspects of stroke (including intimate familiarity with the current state of the art in assessment of its severity and prognosis) combined with sophistication in the area of predictive analytics. Tools developed by our experienced predictive analytics team have entered routine operations in one of the most advanced integrated health care delivery systems in the United States. We were able to characterize the study cohort better than has been done previously. The TSL model included detailed data from index hospitalization not in previous studies on readmission poststroke, including vital signs, laboratory data, patient’s care directive and mRS score at discharge. This study was embedded in a community setting with comprehensive longitudinal information infrastructure. The study cohort was diverse in race-ethnicity and included all ages, not just Medicare-eligible patients. We used prospective validation. All analyses were based on two basic data standards: claims data (functionally, the lowest common denominator) and modern EHR data. This study started from the premise that prediction of a patient with stroke‘s risk for readmission should be automated, presented through a comprehensive EHR, and occur in real time.

There were also limitations with our study. We relied on data collected in the course of routine clinical operations rather than strict research protocols. While our hospitals treat patients with stroke regardless of insurance, our study cohort and analysis were limited to an insured population with access to care to one of the most highly integrated health care delivery systems in the world. The study results may not be applicable to the uninsured population. All of our centers are Joint Commission certified stroke centers with the highest awards from the Get With the Guidelines program. More than 50% of acute ischemic stroke cases qualifying for intravenous thrombolytic were treated within 30 minutes [9]. Therefore, our study results may not be representative of all hospitals caring for patients with ischemic stroke. We did not have baseline functional status and, therefore, could not include a change in functional status to the models. We also did not have indicators for postdischarge functional status. We were unable to include other in-hospital data such as placement of feeding tube or trach due to a high rate of missing data perhaps due to difficulty with pulling correct data electronically. We did not have data on social support or a cognitive function measure at discharge, which may play a significant role in readmission. Our study results are generalizable to an adult population with ischemic stroke who have insurance and access to care who survive the index hospitalization and are treated in certified stroke centers.

In summary, we have tested a more comprehensive predictive model (TSL) for readmission and death within 30 days post ischemic stroke. Having a comprehensive look at the important factors that may influence that decision-making process (socioeconomic status, living arrangement or social support, functional status, code status, cognitive status, etc) would go a long way at improving the performance of a predictive model. Future studies should seek to improve upon our study and evaluate other factors not included here, such as changes from baseline functional status, the volume of brain infarcted, social support, and cognitive status at the time of discharge.