The Karma system is currently undergoing maintenance (Monday, January 29, 2018).
The maintenance period has been extended to 8PM EST.

Karma Credits will not be available for redeeming during maintenance.

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 31.10.18 in Vol 6, No 4 (2018): Oct-Dec

Preprints (earlier versions) of this paper are available at, first published Feb 26, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Impact of Electronic Versus Paper Vital Sign Observations on Length of Stay in Trauma Patients: Stepped-Wedge, Cluster Randomized Controlled Trial

    1Leeds Institute of Health Sciences, Faculty of Medicine and Health, University of Leeds, Leeds, United Kingdom

    2Kadoorie Centre for Critical Care Research and Education, Nuffield Department of Clinical Neurosciences, University of Oxford, Oxford, United Kingdom

    3Centre for Statistics in Medicine, University of Oxford, Oxford, United Kingdom

    4Institute of Biomedical Engineering, Department of Engineering Science, University of Oxford, Oxford, United Kingdom

    Corresponding Author:

    David CW Wong, MEng, DPhil

    Leeds Institute of Health Sciences

    Faculty of Medicine and Health

    University of Leeds

    Worsley Building, Level 10

    Leeds, LS29JT

    United Kingdom

    Phone: 44 113 3439671



    Background: Electronic recording of vital sign observations (e-Obs) has become increasingly prevalent in hospital care. The evidence of clinical impact for these systems is mixed.

    Objective: The objective of our study was to assess the effect of e-Obs versus paper documentation (paper) on length of stay (time between trauma unit admission and “fit to discharge”) for trauma patients.

    Methods: A single-center, randomized stepped-wedge study of e-Obs against paper was conducted in two 26-bed trauma wards at a medium-sized UK teaching hospital. Randomization of the phased intervention order to 12 study areas was computer generated. The primary outcome was length of stay.

    Results: A total of 1232 patient episodes were randomized (paper: 628, e-Obs: 604). There were 37 deaths in hospital: 21 in the paper arm and 16 in the e-Obs arm. For discharged patients, the median length of stay was 5.4 (range: 0.2-79.0) days on the paper arm and 5.6 (range: 0.1-236.7) days on the e-Obs arm. Competing risks regression analysis for time to discharge showed no difference between the treatment arms (subhazard ratio: 1.05; 95% CI 0.82-1.35; P=.68). A greater proportion of patient episodes contained an Early Warning Score (EWS) ≥3 using the e-Obs system than using paper (subhazard ratio: 1.63; 95% CI 1.28-2.09; P<.001). However, there was no difference in the time to the subsequent observation, “escalation time” (hazard ratio 1.05; 95% CI 0.80-1.38; P=.70).

    Conclusions: The phased introduction of an e-Obs documentation system was not associated with a change in length of stay. A greater proportion of patient episodes contained an EWS≥3 using the e-Obs system, but this was not associated with a change in “escalation time.”

    Trial Registration: ISRCTN Registry ISRCTN91040762; (Archived by WebCite at

    JMIR Med Inform 2018;6(4):e10221





    Physiological vital signs are regularly measured in hospitalized patients. Deterioration in vital signs often precedes adverse outcomes [1,2]. However, vital sign alterations can go unrecognized, resulting in treatment delay that may worsen outcomes [3-5]. One method for identifying deterioration is the Early Warning Score (EWS), in which a score is given to each set of recorded vital signs. The overall score, the EWS, is the aggregate of scores assigned to each individual vital sign, depending on its level of abnormality. Higher scores indicate greater physiological abnormality [6].

    Until recently, vital signs and EWS have been recorded on paper observation charts. The shortcomings of paper charts include incorrect score assignment to the vital signs and incorrect calculation of EWS [7-9], difficulty in interpretation [10], and poor compliance with clinical escalation protocols [11]. Electronic methods for recording vital sign observations and EWS, known as e-Obs, are becoming increasingly common [12-14]. e-Obs systems may circumvent many of these highlighted issues by automatically assigning EWS and prompting appropriate clinical response. The facility to display recent observations and scores on a central station may also improve the ability of clinical staff to recognize patient deterioration. However, the clinical impact of such systems is currently unclear, with studies reporting conflicting results [15,16].


    In this study, we prospectively assessed whether the deployment of the VitalPAC e-Obs system (VitalPAC; The Learning Clinic), compared with the paper-based system, changed patients’ hospital length of stay (since the conclusion of this study, The Learning Clinic has been acquired by System C, Maidstone, Kent). Our null hypothesis was that length of stay remained unaltered. A cluster-randomized design was not appropriate as the VitalPAC e-Obs system was to be introduced gradually in the trauma unit. We, therefore, evaluated the intervention as it was being introduced using a stepped-wedge study design.


    Study Design

    We conducted a randomized stepped-wedge interventional study in the two adult inpatient wards of the trauma unit at the John Radcliffe Hospital, Oxford University Hospitals National Health Service Trust. Each ward had 26 inpatient beds. A stepped-wedge study is one in which the intervention is phased into the study population within clusters across successive time periods, with the time being determined by randomization [17]. The full study protocol is available online [18]. Approval for this study was obtained from the National Health Service Research Ethics Committee (REC #11/H0308/11), and the study was registered with ISRCTN (ISRCTN91040762). Informed written consent was sought from all eligible participants after hospital discharge.

    Control and Intervention

    Initially, nursing staff measured vital signs (blood pressure, pulse, oxygen saturation, temperature, respiratory rate, and consciousness level) using spot-check monitors and documented the result on paper. Nurses manually calculated the EWS and recorded the score on paper. The EWS used was the previously published centile-EWS [19], in which a score of 3 or more (EWS≥3) requires urgent intervention.

    The intervention was the VitalPAC e-Obs system, which allowed vital signs to be documented on a hand-held device. The EWS was then automatically calculated, and relevant hospital guidance for escalation was displayed. EWS were also displayed at a central patient ward list that adjusted colors and symbols to prompt nurses to record timely observations according to the hospital protocol.

    Prior to study commencement, VitalPAC was installed and tested and staff were trained to use the system. Refresher teaching was also provided for the paper system, and staff were reminded of local clinical escalation policy. During the training period, the paper system was used throughout the trauma unit, but staff had the opportunity to enter data into a test installation of VitalPAC. Study staff attended the ward daily during the week and once at weekends, providing top-up training throughout the duration of the study.

    Trial Design

    The e-Obs intervention was phased in 12 clusters. Each cluster was a physical zone within the trauma unit consisting of a 4-bed bay or a collection of six 1-bed side rooms. The clusters received the electronic intervention sequentially. A new zone switched from the control to the intervention every 3 weeks on Tuesday at 2 pm.


    All episodes from patients aged ≥16 years admitted to the trauma unit during the study period were considered for eligibility. Episodes were excluded from analysis if the treatment plan for the patient was palliative at admission to the ward.

    Study Data

    Clinical and demographic data on study participants were obtained from an electronic patient record (Casenotes) and from paper medical records. Research nurses collated the following: age, sex, ethnicity, American Society of Anesthesiologists score, reason for admission, and admission method (emergency, elective, between wards) for each study participant.

    We also recorded initial ward and zone on the trauma unit; date and time (hh:mm) of admission into the ward; date, time (hh:mm), and clinician-recorded EWS for the first vital sign observations recorded on the trauma unit; total number of vital sign observations on the unit recorded electronically and on paper; “fit to discharge” date; actual hospital discharge date and time (hh:mm); in-hospital mortality and 30-day mortality following ward admission; and unplanned admission to the intensive care unit (ICU) or a cardiac arrest for each episode.

    Two other electronic sources, Cerner Millenium (Cerner, Kansas City, MO) and Bluespier (Bluespier International, Worcestershire), were used to validate the information. Mortality status at 30 days after admission was verified using data from the National Health Service personal demographic service.


    The primary endpoint was length of stay (the time from admission to the trauma unit until “fit to discharge”). “Fit to discharge” was defined as the first of the following: discharged from the ward to home or alternative care or accepted by social services as a “delayed discharge.” This outcome measure was chosen because some trauma patients were known to have extended stays while waiting for suitable support mechanisms to be put in place outside hospital. Secondary endpoints were mortality (in-hospital and 30-day following ward admission), whether a patient experienced a cardiac arrest or an unplanned ICU admission; the time between observations (length of stay/total number of observations); the time until a patient first scored EWS≥3; and the time between the first observation that scored EWS≥3 and the subsequent observation (“escalation time”).

    Sample Size

    The number of participants was determined by the speed at which zones transitioned from paper to e-Obs. Slower transitions would include more participants for each step, thereby increasing the power of the study. However, clinical staff wished to minimize the concurrent use of multiple systems. Therefore, a clinically accepted transition rate of 1 zone every 3 weeks was chosen, and no sample size calculation was undertaken.


    A random sequence generated using MATLAB (function randperm [20]) determined the order in which zones received e-Obs.

    All zones were recruited and enrolled at baseline and followed for the entire duration of the study. Research nurses administered the sequential assignment to the e-Obs intervention. The nurses visited the ward during each transition to facilitate adherence to the change from paper to e-Obs.

    Patients were allocated to either paper or electronic recording of observations based on the zone to which they were allocated on arrival to the trauma unit. Patients remained with the same recording method even if subsequently moved. Allocation to the initial zone was determined by normal ward practices, which remained unchanged during the study period. Therefore, the allocation ratio could not be determined a priori.


    The randomization sequence was concealed from patients, clinical staff, and all researchers involved in data collection until the day a new zone was due to receive e-Obs. It was not possible to conceal the intervention.

    Statistical Methods

    Episode characteristics, including patient age and length of stay, were summarized by study arm. A time-to-event analysis, Cox proportional hazards regression, was undertaken for length of stay with the competing risk of death in hospital, for death in hospital with discharge from hospital as a competing risk, and for time to first EWS≥3. The intervention arm, step, age, and sex of patients were included as covariates. Subhazard ratios were calculated with respect to paper charts, age>80 years, and male sex. The step was included as a continuous measure of time and as a factor with 13 levels. The study design has two levels, patient and study zone. The SEs of the coefficients in the Cox proportional hazards model were adjusted for the cluster variable, study zone. A similar competing risks Cox regression analysis was undertaken for “escalation time,” censored at 200 hours.

    The binary outcome, death within 30 days of admission, was analyzed using a logistic regression model and the SEs of the coefficients were adjusted for the cluster variable, study zone. The numbers of cardiac arrests and admissions to ICU were reported.

    All analyses were performed on an intention-to-treat basis, with patients analyzed according to the randomization intervention. Furthermore, all analyses were completed for hospital episodes. A patient may have had multiple episodes consisting of distinct admissions to the study wards for unrelated reasons. We assumed a priori that the number of such admissions was small such that each episode may be treated as an independent event. Post hoc, we repeated the analyses on the per-protocol populations.

    All statistical analyses were performed using Stata (Stata Statistical Software: Release 14. College Station, TX; StataCorp LP) [21].


    A total of 1518 admissions to the trauma unit were recorded between August 31, 2011 and May 31, 2012. After excluding 286 episodes that did not meet the study criteria or for which patients had declined consent, 1232 episodes (from 1199 patients) were included for analysis (Figure 1).

    Of the included episodes, 628 were randomized to paper and 604 to e-Obs. Moreover, 873 episodes (paper: 558, e-Obs: 315) had vital sign observations that were fully consistent with the randomized intervention. A further 194 episodes (paper: 32, e-Obs: 162) had over 80% of observations on the allocated intervention. Vital sign observation charts were absent from the paper notes for 13 episodes.

    Baseline characteristics of the study patients are shown in Table 1. Allocation between the study arms was almost equal (paper: 628/1232, 50.97%). There were no significant differences in any of the measured characteristics. In all analyses, the intraclass correlation coefficient for patients within a study area was not significantly different from zero.

    There were 37 deaths in hospital: 21 in the paper arm and 16 in the e-Obs arm. For patients who were discharged, the median length of stay (time from admission to “fit to discharge”) was 5.4 (range: 0.2-79.0) days on the paper arm (607 patients) and 5.6 (range: 0.1-236.7) days on the e-Obs arm (588 patients). Longer time to discharge was associated with greater age, but there was no difference between the treatment arms (Table 2).

    Figure 1. Flow chart for patients’ enrollment in the study. “Escalation Time” is the time between the first observation that scored Early Warning Score (EWS)≥3 and the subsequent observation. ICU: intensive care unit.
    View this figure

    Results from the competing risks Cox regression analysis for time to death in hospital with discharge from hospital as a competing risk are reported in Table 2. There was no difference between the treatment arms.

    There were 45 deaths within 30 days from admission to hospital: 23 in the paper arm and 22 in the e-Obs arm. The results from the logistic regression analysis of the number of deaths within 30 days of admission are reported in Table 2. There was no difference between the treatment arms.

    There were 8 admissions to the ICU from the paper arm and 7 from the e-Obs arm, and there were 2 cardiac arrests in each arm. The median time between observations for those without a delayed discharge was 7.1 (interquartile range [IQR]: 5.0-9.8) hours on paper and 7.0 (IQR: 5.3-9.5) hours on e-Obs.

    Figure 2 shows the Kaplan-Meier estimates of time to first EWS≥3. At least one EWS≥3 was recorded for 278 and 342 episodes in the paper and e-Obs arms, respectively. The number at risk indicates the episodes that had not had an EWS≥3 after {0, 200, 400, 600, 800, and 1000} hours. Numbers in parentheses are those who had an EWS≥3 before the next time point. Moreover, 7 episodes in the e-Obs arm had no recorded observations, while 13 episodes in the paper arm had missing data (observation chart missing from notes). The subhazard ratio for e-Obs with respect to paper from a competing risks Cox regression analysis for time to first EWS ≥3 with death in hospital as a competing risk including age>80 years, step, and sex was 1.63 (95% CI 1.28-2.09; P<.001).

    Figure 3 shows the Kaplan-Meier estimates of escalation time. Inclusion in the analysis was conditional on reaching an EWS of ≥3. The number at risk indicates the episodes that had not yet had an observation after {0, 25, 50,..., 200} hours. Numbers in parentheses are those who had the next set of observations before the next time point. Patients could be censored at any time due to discharge from hospital or death in hospital. Three episodes had the next observation after more than 50 hours, 18 were discharged from hospital, and 3 died in the hospital before a further set of observations. The hazard ratio for e-Obs with respect to paper from a competing risk Cox regression analysis for time to escalation including age>80 years, step, and sex was 1.05 (95% CI 0.80-1.38; P=.70).

    All results presented here are for intention-to-treat analysis. The results of the per-protocol analysis are shown in Multimedia Appendix 1. On per-protocol analysis, we found no difference in “escalation time” or length of stay. Per-protocol in-hospital time to mortality analysis suggested a mortality benefit in favor of e-Obs, but this was not sustained at 30 days.

    Table 1. Patient baseline characteristics (N=1232 episodes).
    View this table
    Table 2. Results from the competing risks regression analysis for time to discharge from hospital with death in hospital as a competing risk, from the competing risks regression analysis for time to death in hospital with discharge from hospital as a competing risk, and the logistic regression for number of deaths within 30 days from admission.
    View this table
    Figure 2. Kaplan-Meier failure estimates for time (hours) from admission until first Early Warning Score (EWS)≥3. e-Obs: electronic recording of vital sign observations.
    View this figure
    Figure 3. Kaplan-Meier failure estimates for time (hours) to next set of observations after the first Early Warning Score (EWS)≥3. e-Obs: electronic recording of vital sign observations.
    View this figure


    Principal Findings

    To our knowledge, this is the first stepped-wedge evaluation of an e-Obs intervention. We found no difference in length of stay (the primary outcome) between paper and e-Obs. A significantly greater proportion of patient episodes contained an EWS≥3 using the e-Obs system than using paper. However, we found no difference in “escalation time.” Per-protocol analysis showed improved in-hospital survival with e-Obs, but this was not shown either on intention-to-treat analysis or for outcome at 30 days. The absence of intraclass correlation indicates that patient groups were well matched at each individual step level.


    Although 86.61% (1067/1232) of patients had over 80% of their observations in the correct format, cross over was unequal. Only 79.0% (477/604) of patients in the electronic observations arm had more than 80% of their observations recorded electronically. The presence of multiple documentation systems in the wards may have led to confusion and suboptimal system use. To avoid this issue, future studies might ensure clear physical separation between zones to prevent study arm contamination. Our follow-up study over 4 hospitals instead considers whole wards as a study zone [22].

    Length of stay is a complex outcome, risking confounding by competing interventions. However, it is an important outcome for patients and their families [23], as well as clinicians and managers, and has been recommended as a potential outcome measure within the National Institute for Health and Care Excellence recommendations for recognizing patient deterioration in hospital [24]. We used it as our primary outcome as delayed detection of deterioration might be expected to increase length of stay even where the delay or deterioration was not sufficient to warrant ICU admission or cause death. Indeed, improved recognition of clinical deterioration is associated with earlier discharge [25] and has previously been used in observational studies assessing e-Obs systems [13,15,26]. To minimize the effect of competing interventions, we used “fit to discharge” rather than actual discharge time and took account of the competing risk of death in our analysis. We included other outcome measures that would be expected to be associated with more extreme delays in recognizing deterioration (death and ICU admission) as secondary outcome measures.


    In this study, more patient episodes contained an EWS≥3 using the e-Obs system than paper. We have previously shown paper documentation errors to be biased toward values lower than the true EWS, particularly when a patient first develops physiological instability [27]. As the groups were otherwise well balanced, it is likely that the result is explained by a bias in paper documentation. As the clinical behavior underlying this bias appears to be related to the actual instability of the patient, it is unclear whether removing the bias will affect patient outcomes [27].

    Despite differences in documented EWS, there was no difference in the timeliness of observations between the two cohorts. This supports the results reported by Hands et al, in which an e-Obs system was introduced to all adult inpatient areas of one hospital [11]. The time of vital sign observations was recorded. They found that observations were more frequently recorded at particular hours in the day, rather than simply responding to escalation algorithms.

    There was also no difference in the primary outcome, patient length of stay from admission to the trauma unit until “fit to discharge” after accounting for potential confounding variables. Previous evaluations of e-Obs systems have focused on process improvements such as data accuracy and speed of documentation [7,28-30]. More recently, longitudinal data have been used to assess the impact of e-Obs on patient mortality via before-and-after analysis, with conflicting results [15,16]. Dawes et al found a 2-day reduction in average length of stay after introduction of the VitalPAC e-Obs system when comparing results in 2010 to those in 2005 [15]. However, further exploration suggested this finding was due to a decreased severity of admission rather than a change in in-hospital care. Jones et al reported reductions in length of stay using the Patientrack e-Obs system [13]. Subsequent correspondence suggested that this may have resulted from changes in discharge processes rather than changes in care resulting from the e-Obs system [31,32].

    A more recent study suggested substantial reductions in hospital mortality in two hospitals that implemented VitalPAC [16]. Although our per-protocol analysis also showed a survival benefit with e-Obs, this was a post hoc analysis of a secondary outcome and was not found in either intention-to-treat analysis or 30-day mortality (analyzing either intention-to-treat or per-protocol groups). We, therefore, do not think great weight should be attached to this outcome, particularly as we did not find any change in observation frequency in those becoming unstable to support the hypothesis that e-Obs caused a change in care.

    The previous before-and-after studies are inherently limited in their ability to account for temporal changes in covariates [33]. Our stepped-wedge design and relatively short total study time reduced the risk of other major changes in practice affecting our outcomes. The stepped-wedge methodology employed here is a practical choice for phased interventions that allows for control of temporal covariates because both control and intervention are active over the whole study period [34]. One disadvantage of the stepped-wedge design is that there is no established consensus on the most appropriate methods of modeling the data [35]. However, if properly analyzed, the quality of evidence is better than that of before-and-after studies and approaches that of randomized controlled trials [36].

    Although e-Obs had no effect on patient outcomes in our study, there are some positive findings. First, the frequency of observations before and after e-Obs remained stable. Furthermore, although the timeliness of observations when patients were physiologically unstable did not improve, they also did not worsen, matching previous reports of e-Obs introduction [13]. These findings suggest that it is possible to introduce an e-Obs system without adversely affecting these ward staff functions. Without an adverse effect, the availability of the vital signs electronically brings the possibility of benefits outside the patients studied, or in the future that may make e-Obs worthwhile.


    The results here are specific to the VitalPAC e-Obs system. This system contains some key features that are available in alternative e-Obs solutions; these include automatic EWS calculation and real-time ward lists [12]. The results are also location specific. While trauma was chosen as a representative specialty that contained a wide range of care, results may not be true in other hospital contexts. Although these factors may reduce the generalizability of results, the results are robust due to the large number of study participants and the stepped-wedge study design.


    The introduction of an electronic system for recording vital sign observations was not associated with reduction in time from admission to the trauma unit until “fit to discharge.” More patient episodes contained an EWS≥3 using the e-Obs system, but this was not associated with a change in “escalation time.”


    We thank the Learning Clinic (System C) for provision of the VitalPAC system and full training for staff during the study. We also thank Jane Cottrell for assistance in developing the study protocol and Jane Boutflower and Gloria Noreen for collecting data during the study. This work was supported by the Oxford National Institute for Health Research Biomedical Research Centre (DFRWAO00).

    Authors' Contributions

    DCWW assisted in collecting the data, managed and refined the data, produced overview statistics, and prepared the manuscript draft. JK collected the data and assisted in preparing the manuscript. JB developed the statistical methodology, computed the statistical analysis of the data, and assisted in preparing the manuscript. LT and PJW designed the research study, provided overall supervision, and revised the manuscript. All authors have read and approved the manuscript.

    Conflicts of Interest

    PJW, DCWW, and LT developed an alternative e-Obs system (SEND) after study completion and prior to paper submission. In February 2017, this system was licensed by the University of Oxford and the Oxford University Hospitals National Health Service Trust to Drayson Technologies. Under the license agreement, PJW, DCWW, and LT may receive future royalties. DCWW and LT have undertaken consultancy for Drayson Technologies. JK and JB declare no conflict of interest. The Learning Clinic had no access to the data or influence on the analysis of the study.

    Multimedia Appendix 1

    Per-protocol analysis.

    PDF File (Adobe PDF File), 296KB

    Multimedia Appendix 2

    CONSORT‐EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 136KB


    1. Cretikos M, Chen J, Hillman K, Bellomo R, Finfer S, Flabouris A. The objective medical emergency team activation criteria: a case-control study. Resuscitation 2007 Apr;73(1):62-72. [CrossRef] [Medline]
    2. Watkinson PJ, Barber VS, Price JD, Hann A, Tarassenko L, Young JD. A randomised controlled trial of the effect of continuous electronic physiological monitoring on the adverse event rate in high risk medical and surgical patients. Anaesthesia 2006 Nov;61(11):1031-1039 [FREE Full text] [CrossRef] [Medline]
    3. McQuillan P, Pilkington S, Allan A, Taylor B, Short A, Morgan G, et al. Confidential inquiry into quality of care before admission to intensive care. BMJ 1998 Jun 20;316(7148):1853-1858 [FREE Full text] [Medline]
    4. Garnacho-Montero J, Aldabo-Pallas T, Garnacho-Montero C, Cayuela A, Jiménez R, Barroso S, et al. Timing of adequate antibiotic therapy is a greater determinant of outcome than are TNF and IL-10 polymorphisms in patients with sepsis. Crit Care 2006;10(4):R111 [FREE Full text] [CrossRef] [Medline]
    5. Chalfin DB, Trzeciak S, Likourezos A, Baumann BM, Dellinger RP, DELAY-ED study group. Impact of delayed transfer of critically ill patients from the emergency department to the intensive care unit. Crit Care Med 2007 Jun;35(6):1477-1483. [CrossRef] [Medline]
    6. Smith GB, Prytherch DR, Schmidt PE, Featherstone PI. Review and performance evaluation of aggregate weighted 'track and trigger' systems. Resuscitation 2008 May;77(2):170-179. [CrossRef] [Medline]
    7. Prytherch DR, Smith GB, Schmidt P, Featherstone PI, Stewart K, Knight D, et al. Calculating early warning scores-a classroom comparison of pen and paper and hand-held computer methods. Resuscitation 2006 Aug;70(2):173-178. [CrossRef] [Medline]
    8. Smith AF, Oakey RJ. Incidence and significance of errors in a patient 'track and trigger' system during an epidemic of Legionnaires' disease: retrospective casenote analysis. Anaesthesia 2006 Mar;61(3):222-228 [FREE Full text] [CrossRef] [Medline]
    9. Subbe CP, Gao H, Harrison DA. Reproducibility of physiological track-and-trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007 Apr;33(4):619-624. [CrossRef] [Medline]
    10. Wilson SJ, Wong D, Clifton D, Fleming S, Way R, Pullinger R, et al. Track and trigger in an emergency department: an observational evaluation study. Emerg Med J 2013 Mar;30(3):186-191. [CrossRef] [Medline]
    11. Hands C, Reid E, Meredith P, Smith GB, Prytherch DR, Schmidt PE, et al. Patterns in the recording of vital signs and early warning scores: compliance with a clinical escalation protocol. BMJ Qual Saf 2013 Sep;22(9):719-726. [CrossRef] [Medline]
    12. Wong D, Bonnici T, Knight J, Morgan L, Coombes P, Watkinson P. SEND: a system for electronic notification and documentation of vital sign observations. BMC Med Inform Decis Mak 2015 Aug 13;15:68 [FREE Full text] [CrossRef] [Medline]
    13. Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Crit Care Resusc 2011 Jun;13(2):83-88. [Medline]
    14. Smith GB, Prytherch DR, Schmidt P, Featherstone PI, Knight D, Clements G, et al. Hospital-wide physiological surveillance-a new approach to the early identification and management of the sick patient. Resuscitation 2006 Oct;71(1):19-28. [CrossRef] [Medline]
    15. Dawes TR, Cheek E, Bewick V, Dennis M, Duckitt RW, Walker J, et al. Introduction of an electronic physiological early warning system: effects on mortality and length of stay. Br J Anaesth 2014 Oct;113(4):603-609 [FREE Full text] [CrossRef] [Medline]
    16. Schmidt PE, Meredith P, Prytherch DR, Watson D, Watson V, Killen RM, et al. Impact of introducing an electronic physiological surveillance system on hospital mortality. BMJ Qual Saf 2015 Jan;24(1):10-20. [CrossRef] [Medline]
    17. Hemming K, Haines TP, Chilton PJ, Girling AJ, Lilford RJ. The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting. BMJ 2015 Feb 06;350:h391 [FREE Full text] [Medline]
    18. Watkinson P, Tarassenko L, Price J, Webb S. nuffield department of clinical medicine. 2011. OTEST: Oxford Track-&-Trigger Electronic System for Trauma Protocol   URL: [accessed 2018-02-21] [WebCite Cache]
    19. Tarassenko L, Clifton DA, Pinsky MR, Hravnak MT, Woods JR, Watkinson PJ. Centile-based early warning scores derived from statistical distributions of vital signs. Resuscitation 2011 Aug;82(8):1013-1018. [CrossRef] [Medline]
    20. The Mathworks, I. MATLAB and statistics toolbox release 2014a. Natick, Massachusetts, United States: The Mathworks, Inc; 2014.   URL: [accessed 2018-10-02] [WebCite Cache]
    21. Banks J. STATA. The Economic Journal 1992 Nov;102(415):1581. [CrossRef]
    22. Bonnici T, Gerry S, Wong D, Knight J, Watkinson P. Evaluation of the effects of implementing an electronic early warning score system: protocol for a stepped wedge study. BMC Med Inform Decis Mak 2016 Feb 09;16:19 [FREE Full text] [CrossRef] [Medline]
    23. Clarke A. Why are we trying to reduce length of stay? Evaluation of the costs and benefits of reducing time in hospital must start from the objectives that govern change. Qual Health Care 1996 Sep;5(3):172-179 [FREE Full text] [Medline]
    24. National Institute for Health and Clinical Excellence. Acutely ill patients in hospital : recognition of and response to acute illness in adults in hospital. (NICE clinical guideline No 50.) 2007. [Medline]
    25. Torsvik M, Gustad LT, Mehl A, Bangstad IL, Vinje LJ, Damås JK, et al. Early identification of sepsis in hospital inpatients by ward nurses increases 30-day survival. Crit Care 2016 Dec 05;20(1):244 [FREE Full text] [CrossRef] [Medline]
    26. Smith MEB, Chiovaro JC, O'Neil M, Kansagara D, Quiñones AR, Freeman M, et al. Early warning system scores for clinical deterioration in hospitalized patients: a systematic review. Ann Am Thorac Soc 2014 Nov;11(9):1454-1465. [CrossRef] [Medline]
    27. Clifton DA, Clifton L, Sandu D, Smith GB, Tarassenko L, Vollam SA, et al. 'Errors' and omissions in paper-based early warning scores: the association with changes in vital signs--a database analysis. BMJ Open 2015 Jul 03;5(7):e007376 [FREE Full text] [CrossRef] [Medline]
    28. Wager KA, Schaffner MJ, Foulois B, Swanson KA, Parker C, Walo H. Comparison of the quality and timeliness of vital signs data using three different data-entry devices. Comput Inform Nurs 2010;28(4):205-212. [CrossRef] [Medline]
    29. Mohammed M, Hayton R, Clements G, Smith G, Prytherch D. Improving accuracy and efficiency of early warning scores in acute care. Br J Nurs 2009;18(1):18-24. [CrossRef] [Medline]
    30. Wong D, Bonnici T, Knight J, Gerry S, Turton J, Watkinson P. A ward-based time study of paper and electronic documentation for recording vital sign observations. J Am Med Inform Assoc 2017 Feb 11. [CrossRef] [Medline]
    31. Smith GB, Prytherch DR, Schmidt PE, Featherstone PI, Meredith P. Bedside electronic capture - can it influence length of stay? Crit Care Resusc 2011 Dec;13(4):281-2; author reply 283. [Medline]
    32. Subbe CP. Bedside electronic capture - methodological concerns. Crit Care Resusc 2011 Dec;13(4):283; author reply 283-283; author reply 284. [Medline]
    33. Goodacre S. Uncontrolled before-after studies: discouraged by Cochrane and the EMJ. Emerg Med J 2015 Jul;32(7):507-508. [CrossRef] [Medline]
    34. Hussey MA, Hughes JP. Design and analysis of stepped wedge cluster randomized trials. Contemp Clin Trials 2007 Feb;28(2):182-191. [CrossRef] [Medline]
    35. Copas AJ, Lewis JJ, Thompson JA, Davey C, Baio G, Hargreaves JR. Designing a stepped wedge trial: three main designs, carry-over effects and randomisation approaches. Trials 2015;16:352 [FREE Full text] [CrossRef] [Medline]
    36. Davey C, Hargreaves J, Thompson JA, Copas AJ, Beard E, Lewis JJ, et al. Analysis and reporting of stepped wedge randomised controlled trials: synthesis and critical appraisal of published studies, 2010 to 2014. Trials 2015;16:358 [FREE Full text] [CrossRef] [Medline]


    e-Obs: electronic recording of vital sign observations
    EWS: Early Warning Score
    ICU: intensive care unit

    Edited by G Eysenbach; submitted 26.02.18; peer-reviewed by P Scott, A Jayatilleke, P Pasquier; comments to author 02.08.18; revised version received 10.08.18; accepted 10.08.18; published 31.10.18

    ©David CW Wong, Julia Knight, Jacqueline Birks, Lionel Tarassenko, Peter J Watkinson. Originally published in JMIR Medical Informatics (, 31.10.2018.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.