RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 27.11.18 in Vol 6, No 4 (2018): Oct-Dec

This paper is in the following e-collection/theme issue:

    Original Paper

    Predictive Modeling of 30-Day Emergency Hospital Transport of Patients Using a Personal Emergency Response System: Prognostic Retrospective Study

    1Philips Research, Eindhoven, Netherlands

    2Partners HealthCare Pivot Labs, Partners HealthCare, Boston, MA, United States

    3Department of Dermatology, Harvard Medical School, Boston, MA, United States

    4Philips Lifeline, Framingham, MA, United States

    5Department of Dermatology, Massachusetts General Hospital, Boston, MA, United States

    6Partners Connected Health, Partners HealthCare, Boston, MA, United States

    Corresponding Author:

    Jorn op den Buijs, PhD

    Philips Research

    High Tech Campus 34

    Eindhoven, 5656 AE


    Phone: 31 631926890



    Background: Telehealth programs have been successful in reducing 30-day readmissions and emergency department visits. However, such programs often focus on the costliest patients with multiple morbidities and last for only 30 to 60 days postdischarge. Inexpensive monitoring of elderly patients via a personal emergency response system (PERS) to identify those at high risk for emergency hospital transport could be used to target interventions and prevent avoidable use of costly readmissions and emergency department visits after 30 to 60 days of telehealth use.

    Objective: The objectives of this study were to (1) develop and validate a predictive model of 30-day emergency hospital transport based on PERS data; and (2) compare the model’s predictions with clinical outcomes derived from the electronic health record (EHR).

    Methods: We used deidentified medical alert pattern data from 290,434 subscribers to a PERS service to build a gradient tree boosting-based predictive model of 30-day hospital transport, which included predictors derived from subscriber demographics, self-reported medical conditions, caregiver network information, and up to 2 years of retrospective PERS medical alert data. We evaluated the model’s performance on an independent validation cohort (n=289,426). We linked EHR and PERS records for 1815 patients from a home health care program to compare PERS–based risk scores with rates of emergency encounters as recorded in the EHR.

    Results: In the validation cohort, 2.22% (6411/289,426) of patients had 1 or more emergency transports in 30 days. The performance of the predictive model of emergency hospital transport, as evaluated by the area under the receiver operating characteristic curve, was 0.779 (95% CI 0.774-0.785). Among the top 1% of predicted high-risk patients, 25.5% had 1 or more emergency hospital transports in the next 30 days. Comparison with clinical outcomes from the EHR showed 3.9 times more emergency encounters among predicted high-risk patients than low-risk patients in the year following the prediction date.

    Conclusions: Patient data collected remotely via PERS can be used to reliably predict 30-day emergency hospital transport. Clinical observations from the EHR showed that predicted high-risk patients had nearly four times higher rates of emergency encounters than did low-risk patients. Health care providers could benefit from our validated predictive model by targeting timely preventive interventions to high-risk patients. This could lead to overall improved patient experience, higher quality of care, and more efficient resource utilization.

    JMIR Med Inform 2018;6(4):e49





    With the worldwide increase in the elderly population [1], chronic diseases and associated health care utilization, such as costly emergency department (ED) visits and subsequent hospitalizations, are also on the rise. In the United States, nearly 40% of patients making ED visits arrived by emergency ambulance transport [2,3], and 80% of unscheduled hospital admissions were through the ED [4]. Preventing avoidable ED visits and admissions among elderly patients is becoming a global priority [5,6], since emergency hospitalizations may be particularly distressing for older people and have been associated with adverse events such as hospital-acquired infections, loss of functional independence, and falls [7].

    Various telehealth programs have been demonstrated to be effective in reducing readmissions, ED visits, and mortality for patients with congestive heart failure (CHF), stroke, and chronic obstructive pulmonary disease (COPD) [8]. For instance, a large nonprofit health care system in the United States introduced a home-based telehealth program that lasts 30 to 60 days after discharge from hospital, targeting the top 5% high-cost patients with multiple chronic conditions [9]. In addition, our recent work unveiled cost-saving opportunities by managing the patients in the lower-cost segments that are at risk of becoming more costly in the long term, such as beyond institutional settings and 30- to 60-day telehealth services [10]. Population health management programs may benefit from data analytics that integrate home monitoring devices to monitor patients after discharge and find out whether they have issues before ED utilization. These devices include personal emergency response systems (PERSs), which can help older adults get immediate assistance when a serious home-based accident occurs and where delayed response may result in preventable ED utilization [11]. Furthermore, our preliminary findings suggested that predictive analysis using PERS data could be useful for identifying patients at high risk for imminent emergency health care utilization [12].

    A PERS service enables older adults to get help in a situation that potentially requires emergency transport by ambulance to the hospital, such as a sudden worsening of their chronic condition or a fall [13,14]. PERS is a widely used wearable technology with a help button that is worn as either a bracelet or a pendant. Patients may press the help button at any time to activate an in-home communication system that connects to a 24/7 call center. The call center associate may contact an informal responder (eg, a neighbor or a family member) or emergency medical services (EMSs) based on the patient’s specific situation and follows up with the patient to confirm that help has arrived. The call center associate records notes from the conversations with the subscribers in an electronic record and classifies the type, situation, and outcome of the case (Figure 1). In combination with user enrollment data, such as demographic, caregiver network, and medical condition data, these case data provide valuable information about the patient’s status.

    Progress in big-data analysis techniques, such as predictive modeling to identify patients at risk of worsening health conditions, may contribute to cost-effectively reducing potentially avoidable health care utilization [15]. Previous efforts in the field of risk prediction include predictive modeling of hospital readmission [16], repeat ED visits [17,18], and the use of specialized discharge services [19]. For example, the LACE index (length of stay; acuity of the admission; comorbidity of the patient, measured with the Charlson comorbidity index score; and emergency department use, measured as the number of visits in the 6 months before admission) was designed for the prediction of death or unplanned readmissions after hospitalization [20], achieving a predictive performance of the area under the receiver operating characteristic curve (AUC) of 0.68. Another model, HOSPITAL (hemoglobin, discharge from an oncology service, sodium level, procedure performed, index type of admission [urgent vs elective], number of admissions in the last year, and length of stay), is a risk score for predicting 30-day potentially avoidable readmission with a performance of AUC=0.72 as evaluated in 9 hospitals in 4 countries [21]. Yet another study used 1-year retrospective electronic health record (EHR) data to predict 30-day ED revisits, with predictive power of AUC=0.70, in a prospective validation cohort [18]. These predictive models rely only on clinical data collected before or during the clinical encounters or at the time of discharge, but not on data from home monitoring postdischarge. In contrast, PERS services collect information while the patient is back at home. This information includes the details—time stamp, type, situation, and outcome—of incidents such as falls, respiratory issues, chest pain, or general pain, as well as other check-in or social calls [13]. Such events may indicate a decline in patient status, and some patients may eventually request emergency transport to the hospital via the PERS service. Thus, patient decline may be captured earlier with PERS–based prediction models than with models based on clinical data. Therefore, the hypothesis of this study is that the medical alert pattern data collected via the PERS service may be used to predict imminent risk for emergency transport to the hospital.

    Figure 1. Overview of the personal emergency response system (PERS) process and data collection.
    View this figure


    The objectives of this study were to (1) develop and validate a predictive model of 30-day emergency hospital transport based on PERS data; and (2) compare the model’s predictions with clinical outcomes derived from the EHR, namely outpatient and inpatient emergency encounters. We used a large, deidentified PERS dataset of more than 580,000 individuals to develop and validate the predictive model. For a subpopulation of 1815 patients, we linked PERS and EHR data so that we could compare PERS–predicted risk of hospital transport with observed clinical outcomes.


    General Overview

    In this study, we followed the guidelines by Luo et al for developing and reporting machine learning predictive models in biomedical research [22]. This was a prognostic retrospective predictive modeling study of emergency hospital transport in the next 30 days. Figure 2 presents an overview of the methodology for predictive model development and evaluation. We used retrospective, deidentified data of 581,675 subscribers of a commercial PERS service (Philips Lifeline, Framingham, MA, USA). We developed and validated the predictive model on cohort 1 using data from 579,860 subscribers. For 1815 subscribers (validation cohort 2), who were patients of the Partners HealthCare at Home (PHH; Partners HealthCare at Home, Inc, Waltham, MA, USA) program in the greater Boston, Massachusetts area, we also collected clinical outcomes in the year following prediction to evaluate how prediction of emergency hospital transport compared with rates of outpatient and inpatient emergency encounters. Patients in the model development cohort and both validation cohorts were mutually exclusive.

    This study was approved by the Internal Committee on Biomedical Experiments of Philips Research (ICBE-2-16049). Furthermore, we obtained approval for linkage of PERS and EHR records from the Partners HealthCare institutional review board.

    Figure 2. Overview of the study design to develop and evaluate the predictive model of emergency hospital transport. AUC: area under the receiver operating characteristic curve; EHR: electronic health record; NPV: negative predictive value; PERS: personal emergency response system; PHH: Partners HealthCare at Home; PPV: positive predictive value.
    View this figure

    Study Cohorts

    Model Development Cohort and Validation Cohort 1

    We included patients if they were subscribers of the Lifeline PERS service on January 1, 2014 (model development cohort) or February 1, 2014 (model validation cohort 1) and were between the ages of 18 and 100 years. We excluded patients if they appeared to be more than 10 years on the service due to the use of a different back-end system prior to 2004 where some patients may have left the service in the meantime without an indication recorded in the database. Also, patients who left the PERS service for any reason in 30 days from the prediction date were excluded from the analysis. Reasons for leaving the service included death, moving to long-term care, and financial reasons. We excluded patients from the model development cohort and validation cohort 1 if they were part of the PPH program, as we included these patients into validation cohort 2. Furthermore, 3.27% (19,886/606,848) of total patients had missing demographic data and were excluded from the analysis.

    The primary PERS data source for model development was the Philips Lifeline database, which contained historical data such as demographics, patients’ living situation and caregiver network, self-reported medical conditions, and medical alert data. We collected data on self-reported medical conditions, including medication allergies, using a custom coding system, which consisted of a drop-down list of common disease categories, such as COPD, CHF, and diabetes. Caregiver network information included the number of responders, the number of people who lived with the patients, and persons to be notified after an incident occurred. Medical alert data from the PERS device included all information gathered during the interactions of the patients with the Lifeline call center associates. Call center associates categorized the calls as incidents or nonincidents. For all calls, the situation (eg, fall, respiratory problems, chest pain, and social call) and the outcome (eg, subscriber okay, responder assistance, and emergency hospital transport) was collected by the associate via custom-made software.

    Validation Cohort 2

    Validation cohort 2 included PERS users who were on the service on February 1, 2014 and received care at PHH, a homecare management service that offers general care as well as specialized services to help patients manage chronic conditions at home. PHH uses a variety of technological innovations to remotely monitor their patients, including the Philips Lifeline PERS, which is a service routinely recommended for elderly or chronically ill patients who are at risk of falls or other health-related emergencies. We excluded patients if they left the PERS service in the 30 days after the prediction date.

    The primary data source for comparison of PERS–based predicted risk for emergency hospital transport with clinical outcomes in validation cohort 2 was the Enterprise Data Warehouse, an electronic medical record data repository of hospitals within the Partners HealthCare System. The data include demographic information, medical conditions, and hospital utilization. We combined longitudinal clinical data from the EHR (from February 1, 2014 to January 31, 2015) for 1815 individuals who met the inclusion criteria with the PERS–based predictive score of emergency hospital transport. All data were deidentified before analysis. It should be noted that, in this study, we did not use EHR data to train the predictive model; rather, we compared PERS–based predictions of 30-day emergency hospital transport with clinical observations from the EHR.

    Data Processing

    We processed PERS data for input into the predictive model using the statistical programming language R [23]. Different database tables were extracted from an operational PERS database, deidentified, and made accessible to the research team.

    One table included subscriber demographics, including age, sex, and region in the United States, subscription type, and enrollment information. A few test accounts, used for demonstration purposes and training of call center agents, were removed prior to the predictive analysis. We used enrollment information to determine which subscribers were active on the prediction date. Subscribers with missing enrollment dates (79,764/1,221,073, 6.53% of all subscribers in the initial data extract) were already deactivated or were pending a PERS installation and were therefore excluded from further analysis. We derived time on the PERS service as the number of days between enrollment and prediction dates.

    One table consisted of the subscribers’ caregiver network information. This included the number of responders (eg, son, daughter, or neighbor) and number of emergency service providers (eg, ambulance or fire department) listed by the subscriber at the time of enrollment.

    One table included self-reported medical conditions and medication allergies. Medical conditions and medication allergies were recorded at enrollment by the call center agent via a custom-made drop-down list for the purpose of informing EMSs in case of an incident. We extracted the 50 most frequently occurring medical conditions and medication allergies from the database for use in the prediction model. Up to 3 of the most common medical conditions and medication allergies were listed for various categories (Table 1).

    Table 1. The most common self-reported medical conditions and medication allergies by personal emergency response system subscribers per category (N=581,675).
    View this table

    One table consisted of up to 2 years of historical PERS service use data, which totaled to more than 25 million data points. We categorized cases by case type, situation, and outcome (Table 2). For each case, we constructed 2 features for input to the predictive model, namely frequency and recency. Frequency was the sum of cases experienced by the subscriber prior to the prediction date. Recency was the number of days between the most recent case prior to the prediction date and the prediction date.

    Table 2. Case types, situations, and outcomes for which frequency and recency features were derived for input into the predictive model. Examples are given per category.
    View this table

    We derived the dependent variable—having 1 or more emergency hospital transports in the 30 days from the prediction date—from the PERS service use data. As the prediction problem was treated as a classification problem, the dependent variable was rendered binary, with 1 if the patient had 1 or more emergency hospital transports in the 30 days following the prediction date and 0 otherwise.

    After preprocessing, we merged the data into 1 large table with 129 columns representing predictive model features and 1 column for the dependent variable. Each of the rows corresponded to 1 PERS user. For 1815 PERS users in the PHH program, clinical outcomes from the EHR were available, and we assigned these to validation cohort 2. We randomly assigned the remaining PERS users on a 1:1 basis to the model development cohort and the validation cohort 1.

    Predictive Model of Emergency Hospital Transport

    We then developed a predictive model using a boosted regression trees approach called extreme gradient boosting within R [24]. We chose extreme gradient boosting because it was adopted by more than half of the winning competitions on the Kaggle platform for data science competitions in 2015 [24]. This methodology was also carried out within the commercial PERS–based prediction system CareSage (Philips Lifeline). During initial model development, we also considered logistic regression as a candidate modeling technique but it resulted in significantly lower AUC values in the validation cohort.

    The frequency and recency features based on medical alert pattern data are highly skewed. While this may pose a problem for certain predictive modeling methods, the tree-based method we used did not make any assumptions on normality of the features. Monotonic transformations of the features to render them more normal did not affect the predictive performance of the models. We determined variable importance according to the gain, a measure of the relative contribution of the corresponding variable to the predictive model, calculated by taking the improvement in accuracy brought by a variable to the branches it is on. We reduced the number of features by selecting only those with non–zero gain values for the final model, resulting in a total of 121 features in the final model. The boosted regression model also involved tuning hyperparameters of the learning algorithm, such as the number of trees, the maximum depth of the trees, and the learning rate. We achieved this optimization using 5-fold cross-validation on the model development set with the optimization metric determined by the AUC in the test fold.

    Predictive Model Evaluation

    We evaluated the discriminatory accuracy of the predictive models using the AUC, which indicates the probability of the predictive model ranking a randomly selected patient with 30-day emergency transport higher than a randomly selected patient without the event. Furthermore, the negative predictive value (NPV) is the percentage of patients not having emergency transport in the group classified as negative, while the positive predictive value (PPV) indicates the percentage of patients having emergency transport in the group classified as positive. We varied the threshold for classifying patients as positive using risk scores above the 90th, 95th, and 99th percentiles such that 10%, 5%, and 1%, respectively, of patients were classified as high risk. For these thresholds, we computed the NPV, PPV, sensitivity, specificity, and accuracy. We derived 95% confidence intervals for performance metrics using a stratified bootstrapping method with 1000 bootstrap replicates.

    While we used AUC to assess the ability of the predictive model to discriminate between patients with and without transport, regularization, as done in extreme gradient boosting, may have created bias in the predictive model [22]. Therefore, we also checked the model for calibration—the agreement between predictions made by the model and the outcome—by plotting the observed outcome in ranges of the predicted probabilities (0%-20%, 20%-40%, …, 80%-100%). We also tested goodness-of-fit using the Hosmer-Lemeshow test. This is a systematic way of assessing whether the observed outcomes match the predicted probabilities in subgroups of the model population.

    Statistical Analysis

    We tested differences in age between the model development and 2 validation cohorts using analysis of variance followed by Tukey post hoc test. Pairwise comparison between pairs of proportions with Holm correction for multiple testing was applied to the patients’ sex, self-reported medical conditions, and 30-day emergency transport.

    To determine how PERS predicted risk for 30-day emergency hospital transport compared with clinical outcomes, we derived the rates of emergency outpatient and inpatient encounters from the EHR at 30, 60, 90, 180, and 365 days following prediction for validation cohort 2. We split the cohort into low, medium, and high risk according to the risk thresholds that corresponded to the 0 to 50th percentile, 51st to 95th percentile, and greater than 95th percentile. The numbers of emergency encounters in these 3 risk groups were normalized as the number of emergency encounters per 100 patients. Emergency encounters were statistically compared using a pairwise Wilcoxon rank sum test.


    Patient Characteristics and Prevalence of the Dependent Variable

    Table 3 presents the characteristics of the model development and both validation cohorts. PERS users were on average 81 years old and most were female (234,817/290,434, 80.85%) in the model development cohort and validation cohort 1. There was no statistically significant difference in age between the model development and validation cohort 1. Average age was slightly, but statistically significantly, younger in validation cohort 2 than in the model development cohort.

    The distribution of the number of self-reported conditions was different in validation cohort 2, with only 6.89% (125/1815) of users not reporting any conditions, compared with 20.58% (59,769/290,434) in the model development cohort. One reason may be that these were patients receiving care for 1 or more specific conditions through a home health care program. Regarding the dependent variable of having 1 or more emergency hospital transports in 30 days, 2.20% (40/1815) of patients had emergency hospital transport in the 30 days following the prediction date in validation cohorts 1 and 2. This was not significantly different from the model development cohort (6686/290,434, 2.30%). The ratio of positive to negative classes was 0.024 for the model development cohort and 0.023 for both validation cohorts. According to the EHR data, 509/1815 patients of validation cohort 2 (28.04%) had 1 or more emergency encounters in the year following the prediction date.

    Model Performance

    Table 4 details the performance of the predictive model in both validation cohorts for various risk thresholds. The AUC for emergency hospital transport in 30 days was 0.779 (95% CI 0.774-0.785) in validation cohort 1 and 0.766 (95% CI 0.686-0.845) in validation cohort 2. Nonsignificant Hosmer-Lemeshow test results (P=.99 for validation cohort 1 and P=.78 for validation cohort 2) showed that predicted probabilities matched observed outcomes.

    Table 3. Patient characteristics and prevalence of the dependent variable in the model development and validation cohorts.
    View this table
    Table 4. Performance of the predictive model evaluated by the area under the receiver operating characteristic curve (AUC), negative predictive value (NPV), positive predictive value (PPV), sensitivity, specificity, and accuracy using the 90th, 95th, and 99th percentiles as thresholds in the 2 validation cohorts.
    View this table

    PPVs were on the low side due to the low prevalence of 30-day emergency hospital transport, which was around 2.2% in both validation cohorts (Table 3). By increasing the prediction score threshold, PPV increased but at the expense of decreased sensitivity. In validation cohort 1, at a risk score threshold corresponding to the 90th percentile, the predictive model identified 43.8% (95% CI 42.5%-45.0%) of the patients who had emergency transport in the 30 days following the prediction date (sensitivity); however, only 9.6% (95% CI 9.3%-9.9%) of flagged patients had emergency transport in the following 30 days (PPV) at this threshold. At thresholds corresponding to the 95th and 99th percentiles, the sensitivity dropped to 30.5% (95% CI 29.3%-31.7%) and 11.5% (95% CI 10.7%-12.3%), respectively, while the PPV increased to 13.5% (95% CI 13.1%-14.0%) and 25.5% (95% CI 24.1%-27.2%), respectively. This analysis using different thresholds illustrates the trade-off between sensitivity and PPV, that is, trying to find as many positive cases as possible with an acceptable false-positive rate. A similar trade-off between sensitivity and PPV was also observed in validation cohort 1.

    The predictive model produced a probability (from 0% to 100%) for each patient to assess the risk of 30-day emergency hospital transport. The actually observed percentage of patients with emergency hospital transport increased as the predicted probabilities increased (Figure 3). At predicted probabilities between 80% and 100%, 80% (4/5) of patients had 30-day emergency hospital transport in validation cohort 1.

    Predictor Variables

    Table 5 provides the number of predictors and total gain, a measure of predictability, for each broad category of predictors. Predictors from the medical alert data formed the most important predictor category for the predictive model, as they accounted for 87.7% of the total gain. A total of 8 predictor variables with zero gain did not contribute to predictability. These included certain uncommon medication allergies and self-reported medical conditions.

    The predictive model of 30-day emergency hospital transport included 121 variables with nonzero values for the gain. Figure 4 shows the 5 most important predictors for each category. For self-reported medical conditions, COPD, CHF, and heart conditions are among the 5 most important predictors. Other important predictors include age, sex, and the number of responders.

    Figure 3. Observed percentage of patients needing 30-day emergency hospital transport versus model-predicted probability in validation cohort 1.
    View this figure
    Table 5. Number of predictors and total gain per predictor category.
    View this table

    Comparison With Clinical Outcomes

    We then used the predictive model of 30-day emergency hospital transport to segment validation cohort 2 into high risk (>95th percentile), medium risk (51st-95th percentile), and low risk (≤50th percentile) after ranking them according to the predicted probabilities of 30-day emergency hospital transport. We normalized the number of emergency encounters (both outpatient visits and inpatient admissions) in the year following the prediction date to 100 patients (Figure 5). Compared with emergency encounters in the low-risk group, patients in the high-risk group had significantly more emergency encounters at 30 (P=.001), 60 (P<.001), 90 (P<.001), 180 (P<.001), and 365 (P<.001) days. The medium-risk group had significantly more emergency encounters at 90 (P=.01), 180 (P<.001), and 365 (P<.001) days. After 365 days, there were 3.9 times more emergency encounters in the high-risk group than in the low-risk group.

    Figure 4. The 5 most important variables in the predictive model for 3 categories of predictors: predictors derived from medical alert data, self-reported medical conditions, and other predictors. Predictor importance as measured by the gain is reported for validation cohort 1. COPD: chronic obstructive pulmonary disease; PERS: personal emergency response system.
    View this figure
    Figure 5. Emergency encounters per 100 patients (pts) in low-, medium-, and high-risk groups in the year following the prediction date. Data shown are for validation cohort 2. *P<.05 compared with low risk, pairwise Wilcoxon rank sum test.
    View this figure


    Principal Findings

    Retrospective validation demonstrated the effectiveness of the PERS medical alert data-based predictive model of 30-day emergency hospital transport in identifying patients at risk of hospital transport with good discrimination (AUC=0.779). A goodness-of-fit test and calibration plot indicated that the model-predicted probabilities matched with observed outcomes across ranges of predicted risk. We evaluated the trade-off between finding many true-positive cases (ie, a high sensitivity) and reducing the number of false-positives (ie, a high PPV) by varying the threshold to classify patients as positive. At a risk threshold corresponding to the 99th percentile, 1 out of 4 patients flagged at high risk needed hospital transport in the next 30 days (PPV=25.5%), while at this threshold the sensitivity was rather low at 11.5%. PPV was nearly 12 times higher than the prevalence of 30-day emergency transport in validation cohort 1, which was 2.20%. The trade-off between sensitivity and PPV was also reported by other predictive modeling studies of emergency health care utilization [18,25].

    Recent studies have reported on the development and validation of predictive models that can be used to systematically identify individuals at high risk for an unfavorable outcome in a hospital setting [26,27]. Such risk assessment models have great potential to inform treatment decisions and improve the quality of care delivered to patients [26-28]. As innovative connected health technologies and value-based care policy influence the evolution of geriatric models of care to deliver more individualized, multifaceted management strategies, home health programs for community-dwelling seniors may benefit from enhanced risk assessment of patients. For instance, a 32-center randomized controlled trial of the guided care model used health care encounter–based predictive modeling software to identify the 25% of patients with the highest risk of needing complex health care in the coming year. The study provided them with geriatric assessment and specialized support, resulting in significantly reduced health care utilization by home health care patients compared with a group that followed usual care [29]. By using remotely collected PERS data to predict the risk of emergency health care utilization, our predictive model of 30-day emergency hospital transport presents a unique opportunity to efficiently allocate limited health care resources to patients who need them the most and thereby reduce costs associated with excessive use.

    This predictive modeling study was, to the best of our knowledge, unique, as it used PERS data to predict emergency hospital transport. Most state laws in the United States require that patients in need of emergency medical care must be taken to the nearest appropriate health care facility capable of treating the patient, which may be an out-of-network facility. Furthermore, in case of ED crowding [30], ambulance diversion to other potentially out-of-network facilities may be requested. In both cases, the resulting ED visit may not be captured in the EHR of the in-network organization. Therefore, predictive models of emergency health care utilization developed on EHR data of a particular health care provider may be missing outcome data. We minimized this limitation in our study, since response agents from the PERS service recorded information about any ambulance transport after an incident that required help from an EMS in the electronic case record of the user.

    Linkage of PERS and EHR data in a cohort of 1815 PERS users who were patients of an accountable care organization in the greater Boston area group enabled comparison of PERS–based predicted risk of 30-day emergency hospital transport with emergency encounters derived from the clinical EHR. The patients in the high-risk group—that is, those above the 95th percentile range of risk for emergency transport—also had 3.9 times more emergency encounters per 100 patients in the year following the prediction date than did patients classified as having low risk. Here we specifically analyzed rates of health care encounters, as these are the events that could ultimately be avoided with the appropriate interventions. These results suggest that prediction of emergency health care utilization based on PERS data may be a good alternative to EHR-based prediction models, which could be especially helpful for continuous monitoring of patients after discharge and where patients have missing or limited EHR records.

    Our previous study on health care utilization by PERS users indicated that 21% of emergency admissions were considered potentially avoidable [10]. Therefore, we believe that prediction of risk for emergency transport, in combination with appropriate interventions, could potentially reduce emergency hospitalizations. Clinical and financial implications of predictive models largely depend on how well case managers and health professionals can integrate risk prediction of patients into clinical workflows. It is vital to have a detailed guideline that clarifies how the algorithm will inform care [15]. An example of such an approach is a predictive model to detect impending deterioration of patients outside the intensive care unit [31], which was implemented in routine clinical care in 2 community hospitals [32]. In an ongoing randomized clinical trial, we are developing workflows that integrate daily PERS–based risk of 30-day emergency hospital transport with care pathways [33]. In this randomized clinical trial, we are using the predictive model described herein to predict patients’ risk for 30-day emergency transport, followed by a nurse assessment and tailored interventions for high-risk patients. The number of patients who will ultimately benefit from a combination of prediction and intervention will depend on various factors, including the population size and the prevalence of emergency health care utilization, the performance of the predictive model and the risk threshold above which patients are considered to be high risk, and the efficacy of the interventions provided to high-risk patients.


    This study had a few limitations. The PERS population is mostly old and primarily female, and the service is predominantly privately paid for by patients and not covered by their health insurance. This may limit the generalizability of the study to older women who can afford the service. Interpretation of the predictive model may have been influenced by confounding of unobserved variables, including when and where users wore the PERS device [34]. While the second objective of the study was to validate the PERS–based prediction model in a specific health system, we believe it is more broadly applicable, since the prediction model was developed on PERS–data only and did not rely on specific inputs from the EHR.

    More PERS subscribers in validation cohort 2 self-reported on medical conditions than in the development and validation cohorts. It is likely that this cohort was a biased patient group with higher risk and more medical conditions, as they were receiving care from a health care provider. Given that the AUC for 30-day transport was not significantly lower in validation cohort 2, we do not expect that this affected the predictions. Furthermore, patients with emergency transport in validation cohort 2 may not have ended up in the ED of one of the hospitals of the Partners HealthCare network, but in a different hospital system, such that emergency health care utilization was not recorded in the EHR data that we used. Additionally, patients may have initiated emergency hospital transport outside of the PERS service. Both above-mentioned limitations may have affected the correlation between PERS–based risk for emergency hospital transport and emergency encounters derived from the EHR.


    This study showed that remotely collected patient data using a PERS service can be used to predict 30-day hospital transport. Furthermore, linking these data to clinical observations from the EHR showed that predicted high-risk patients had nearly four times higher rates of emergency encounters in the year following the prediction date compared with low-risk patients. Health care providers could benefit from our validated predictive model by estimating the risk of 30-day emergency hospital transport for individual patients and target timely preventive interventions to high-risk patients. We are testing this hypothesis in a randomized clinical trial where risk predictions are combined with a stepped intervention pathway. This approach could lead to overall improved patient experience, higher quality of care, and more efficient resource utilization. Future studies should explore the impact of combined EHR and PERS data on predictive accuracy.

    Authors' Contributions

    JodB, MS, SG, LS, and SA designed the research. JodB, MS, and SG conducted the analyses. JF, LS, JK, and KJ provided feedback on analyses and interpretation of results. JodB, MS, SG, NF, JF, LS, and SA wrote the paper, and JodB had primary responsibility for the final content. All authors read and approved the final manuscript.

    Conflicts of Interest

    Philips funded the study. JodB, MS, and LS are employed by Philips.


    1. Suzman R, Beard JR, Boerma T, Chatterji S. Health in an ageing world--what do we know? Lancet 2015 Feb 07;385(9967):484-486. [CrossRef] [Medline]
    2. Ellis G, Marshall T, Ritchie C. Comprehensive geriatric assessment in the emergency department. Clin Interv Aging 2014 Nov;9:2033-2043 [FREE Full text] [CrossRef] [Medline]
    3. Albert M, McCaig LF, Ashman JJ. Emergency department visits by persons aged 65 and over: United States, 2009-2010. NCHS Data Brief 2013 Oct(130):1-8 [FREE Full text] [Medline]
    4. Kocher KE, Dimick JB, Nallamothu BK. Changes in the source of unscheduled hospitalizations in the United States. Med Care 2013 Aug;51(8):689-698. [CrossRef] [Medline]
    5. Iwata M, Kuzuya M, Kitagawa Y, Iguchi A. Emergency department use by nonagenarian patients. Geriatr Gerontol Int 2006 Mar;6(1):25-31. [CrossRef]
    6. Downing A, Wilson R. Older people's use of accident and emergency services. Age Ageing 2005 Jan;34(1):24-30. [CrossRef] [Medline]
    7. de Vries EN, Ramrattan MA, Smorenburg SM, Gouma DJ, Boermeester MA. The incidence and nature of in-hospital adverse events: a systematic review. Qual Saf Health Care 2008 Jun;17(3):216-223 [FREE Full text] [CrossRef] [Medline]
    8. Bashshur RL, Shannon GW, Smith BR, Alverson DC, Antoniotti N, Barsan WG, et al. The empirical foundations of telemedicine interventions for chronic disease management. Telemed J E Health 2014 Sep;20(9):769-800 [FREE Full text] [CrossRef] [Medline]
    9. Dahl D, Reisetter JA, Zismann N. People, technology, and process meet the triple aim. Nurs Adm Q 2014;38(1):13-21. [CrossRef] [Medline]
    10. Agboola S, Simons M, Golas S, op den Buijs J, Felsted J, Fischer N, et al. Health care cost analyses for exploring cost savings opportunities in older patients: longitudinal retrospective study. JMIR Aging 2018 Aug 01;1(2):e10254. [CrossRef]
    11. Bakk L. Does PERS and falls prevention information reduce stress, anxiety, falls risk, hospital admission and emergency room use for older adults on the wait list for home-and community-based services? Final Report. Southfield, MI: Area Agency on Aging 1-B; 2011 May. Accessed 2018-10-09   URL: [accessed 2018-10-09] [WebCite Cache]
    12. op den Buijs J, Smits T, Pijl M, Simons M, Schertzer L. Predictive modeling of emergency hospital transport using medical alert pattern data: retrospective cohort study. iProceedings 2015 Oct 27;1(1):e19. [CrossRef]
    13. Agboola S, Golas S, Fischer N, Nikolova-Simons M, op den Buijs J, Schertzer L, et al. Healthcare utilization in older patients using personal emergency response systems: an analysis of electronic health records and medical alert data. BMC Health Serv Res 2017 Apr 18;17(1):282. [CrossRef] [Medline]
    14. Redd JL, Zura RD, Tanner AE, Walk EE, Wu MM, Edlich RF. Personal emergency response systems. J Burn Care Rehabil 1992;13(4):453-459. [Medline]
    15. Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff (Millwood) 2014 Jul;33(7):1123-1131. [CrossRef] [Medline]
    16. Kansagara D, Englander H, Salanitro A, Kagen D, Theobald C, Freeman M, et al. Risk prediction models for hospital readmission: a systematic review. JAMA 2011 Oct 19;306(15):1688-1698 [FREE Full text] [CrossRef] [Medline]
    17. Meldon SW, Mion LC, Palmer RM, Drew BL, Connor JT, Lewicki LJ, et al. A brief risk-stratification tool to predict repeat emergency department visits and hospitalizations in older patients discharged from the emergency department. Acad Emerg Med 2003 Mar;10(3):224-232 [FREE Full text] [Medline]
    18. Hao S, Jin B, Shin AY, Zhao Y, Zhu C, Li Z, et al. Risk prediction of emergency department revisit 30 days post discharge: a prospective study. PLoS One 2014;9(11):e112944 [FREE Full text] [CrossRef] [Medline]
    19. Holland DE, Harris MR, Leibson CL, Pankratz VS, Krichbaum KE. Development and validation of a screen for specialized discharge planning services. Nurs Res 2006;55(1):62-71. [Medline]
    20. van Walraven C, Dhalla IA, Bell C, Etchells E, Stiell IG, Zarnke K, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ 2010 Apr 06;182(6):551-557 [FREE Full text] [CrossRef] [Medline]
    21. Donzé JD, Williams MV, Robinson EJ, Zimlichman E, Aujesky D, Vasilevskis EE, et al. International validity of the HOSPITAL score to predict 30-day potentially avoidable hospital readmissions. JAMA Intern Med 2016 Apr;176(4):496-502 [FREE Full text] [CrossRef] [Medline]
    22. Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res 2016 Dec 16;18(12):e323 [FREE Full text] [CrossRef] [Medline]
    23. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Foundation; 2015.
    24. Chen T, Guestrin C. XGBoost: a scalable tree boosting system [arXiv:1603.02754]. 2016 Mar 09.   URL: [accessed 2018-10-08] [WebCite Cache]
    25. Billings J, Dixon J, Mijanovich T, Wennberg D. Case finding for patients at risk of readmission to hospital: development of algorithm to identify high risk patients. BMJ 2006 Aug 12;333(7563):327 [FREE Full text] [CrossRef] [Medline]
    26. Hippisley-Cox J, Coupland C. Development and validation of risk prediction algorithms to estimate future risk of common cancers in men and women: prospective cohort study. BMJ Open 2015 Mar 17;5(3):e007825-e007825. [CrossRef] [Medline]
    27. Hippisley-Cox J, Coupland C, Brindle P. Development and validation of QRISK3 risk prediction algorithms to estimate future risk of cardiovascular disease: prospective cohort study. BMJ 2017 May 23:j2099. [CrossRef] [Medline]
    28. LaValley MP, Lo GH, Price LL, Driban JB, Eaton CB, McAlindon TE. Development of a clinical prediction algorithm for knee osteoarthritis structural progression in a cohort study: value of adding measurement of subchondral bone density. Arthritis Res Ther 2017 May 16;19(1):95 [FREE Full text] [CrossRef] [Medline]
    29. Boyd CM, Reider L, Frey K, Scharfstein D, Leff B, Wolff J, et al. The effects of guided care on the perceived quality of health care for multi-morbid older persons: 18-month outcomes from a cluster-randomized controlled trial. J Gen Intern Med 2010 Mar;25(3):235-242 [FREE Full text] [CrossRef] [Medline]
    30. Richardson LD, Asplin BR, Lowe RA. Emergency department crowding as a health policy issue: past development, future directions. Ann Emerg Med 2002 Oct;40(4):388-393. [Medline]
    31. Escobar GJ, LaGuardia JC, Turk BJ, Ragins A, Kipnis P, Draper D. Early detection of impending physiologic deterioration among patients who are not in intensive care: development of predictive models using data from an automated electronic medical record. J Hosp Med 2012 Mar;7(5):388-395. [CrossRef] [Medline]
    32. Dummett BA, Adams C, Scruth E, Liu V, Guo M, Escobar GJ. Incorporating an early detection system into routine clinical practice in two community hospitals. J Hosp Med 2016 Nov;11 Suppl 1:S25-S31 [FREE Full text] [CrossRef] [Medline]
    33. Palacholla RS, Fischer NC, Agboola S, Nikolova-Simons M, Odametey S, Golas SB, et al. Evaluating the impact of a web-based risk assessment system (CareSage) and tailored interventions on health care utilization: protocol for a randomized controlled trial. JMIR Res Protoc 2018 May 09;7(5):e10045 [FREE Full text] [CrossRef] [Medline]
    34. Stokke R. The personal emergency response system as a technology innovation in primary health care services: an integrative review. J Med Internet Res 2016 Jul 14;18(7):e187 [FREE Full text] [CrossRef] [Medline]


    AUC: area under the receiver operating characteristic curve
    CHF: congestive heart failure
    COPD: chronic obstructive pulmonary disease
    ED: emergency department
    EHR: electronic health record
    EMS: emergency medical service
    NPV: negative predictive value
    PERS: personal emergency response system
    PHH: Partners HealthCare at Home
    PPV: positive predictive value

    Edited by G Eysenbach; submitted 22.01.18; peer-reviewed by S Khairat, T Guetterman, SH Kang; comments to author 21.03.18; revised version received 20.07.18; accepted 07.08.18; published 27.11.18

    ©Jorn op den Buijs, Mariana Simons, Sara Golas, Nils Fischer, Jennifer Felsted, Linda Schertzer, Stephen Agboola, Joseph Kvedar, Kamal Jethwani. Originally published in JMIR Medical Informatics (, 27.11.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.