Published on in Vol 8, No 11 (2020): November

Temporal Design Patterns for Digital Phenotype Cohort Selection in Critical Care: Systematic Literature Assessment and Qualitative Synthesis

Temporal Design Patterns for Digital Phenotype Cohort Selection in Critical Care: Systematic Literature Assessment and Qualitative Synthesis

Temporal Design Patterns for Digital Phenotype Cohort Selection in Critical Care: Systematic Literature Assessment and Qualitative Synthesis

Original Paper

1School of Computing and Information Systems, Centre for Digital Transformation of Health, University of Melbourne, Melbourne, Australia

2Department of Internal Medicine, School of Medicine, Pontificia Universidad Catolica de Chile, Santiago, Chile

3Department of Biomedical Informatics, Clinica Alemana, Santiago, Chile

4Instituto de Ciencias e Innovación en Medicina, Facultad de Medicina Clínica Alemana, Universidad del Desarrollo, Santiago, Chile

Corresponding Author:

Daniel Capurro, MD, PhD

School of Computing and Information Systems

Centre for Digital Transformation of Health

University of Melbourne

Room 3.24, Level 3, Doug McDonnel (Building 168)

Parkville Campus

Melbourne, 3010


Phone: 61 8344 4504


Background: Inclusion criteria for observational studies frequently contain temporal entities and relations. The use of digital phenotypes to create cohorts in electronic health record–based observational studies requires rich functionality to capture these temporal entities and relations. However, such functionality is not usually available or requires complex database queries and specialized expertise to build them.

Objective: The purpose of this study is to systematically assess observational studies reported in critical care literature to capture design requirements and functionalities for a graphical temporal abstraction-based digital phenotyping tool.

Methods: We iteratively extracted attributes describing patients, interventions, and clinical outcomes. We qualitatively synthesized studies, identifying all temporal and nontemporal entities and relations.

Results: We extracted data from 28 primary studies and 367 temporal and nontemporal entities. We generated a synthesis of entities, relations, and design patterns.

Conclusions: We report on the observed types of clinical temporal entities and their relations as well as design requirements for a temporal abstraction-based digital phenotyping system. The results can be used to inform the development of such a system.

JMIR Med Inform 2020;8(11):e6924



The increasing costs of health care [1] and the rapid advance of new discoveries create the need for streamlining the identification of effective health interventions. The evidence-based clinical practice paradigm promotes the generation of such knowledge through high-quality randomized controlled trials and systematic reviews [2]. However, when we consider the amount of resources required to conduct a randomized controlled trial [3], alternative ways to assess the effectiveness of clinical interventions become attractive.

The broad adoption of electronic health records (EHRs) [4] allows researchers to analyze routinely collected electronic clinical data to conduct comparative effectiveness research. A health care system that systematically analyzes clinical data to generate and test hypotheses should be able to learn from itself, becoming a learning health care system [5]. However, converting a traditional health care system into a learning one faces several organizational, societal, and data-related barriers.

“Good data” is a relative concept [6], because it depends on who the user is and what the data are being used for. When EHR data are collected primarily for direct patient care and not with the explicit objective of generating knowledge, a majority of the captured information is stored as free text or other types of unstructured format, limiting its reuse potential. Our group has estimated that 75% of all data elements required for calculating clinical quality measures are not available as structured and computable database fields [7]. Similar results have been found about the clinical information required for clinical trial or cohort eligibility criteria [8]. The combination of data and rules to specify the latter are denominated a phenotyping algorithm [9]; digital phenotypes are the cornerstone of generating new knowledge from routinely collected clinical data and of a learning health care system.

The value of structured data lies in its capacity of being computed without major processing, therefore several attempts have been made to overcome the lack of structured clinical data. A review by Shivade et al [10] reported that the most frequently used methods to automatically identify patient cohorts based on EHR phenotypes were rule-based systems, natural language processing, and machine learning techniques. In this review a majority of studies involved the use of diagnostic codes to select eligible patients. However, although wide variations are seen, diagnostic codes frequently present poor sensitivity and specificity to accurately determine patients’ conditions [11].

Despite current advances in the area, cohort building systems require a significant amount of effort to develop and test and, in real scenarios, the most commonly used strategy to deal with limited EHR data quality is to use a combination of simple rules and manual verification of clinical data from patient records [12]. Thus, the field is still open to new and complementary approximations to identify patient cohorts based on digital phenotypes.

Clinical researchers face many barriers when querying clinical databases to find patients that match a specific cohort definition. One problem is that querying clinical databases is a complex task requiring multiple interactions between clinical researchers and database experts. Among those complexities, inclusion criteria frequently define temporal patterns of clinical events, which need convoluted temporal database queries [13]. This is needed in up to 40% of studies [8]. Finding patients that meet certain temporal patterns of clinical events could be both a barrier—when systems that do not easily support this feature are not available—and a very powerful tool to accurately retrieve patient cohorts based on these temporal digital phenotypes. However, systems that easily support this feature are not readily available.

In this study, we systematically reviewed the critical care literature to characterize the temporal representation of inclusion criteria, interventions, and outcomes, used by clinical researchers when designing a clinical study. The product of this review is a set of basic temporal entities, temporal relations, and the resulting temporal phenotype design patterns. The results can be used to inform the design of temporal abstraction-based digital phenotyping systems.

Data Source

We conducted a systematic literature review of published articles in the critical care domain. Using the Web of Science Journal Citation Reports, we selected the top 5 critical care journals according to their impact factor. Paired reviewers (MB, CD, JT, and JS) manually reviewed all publications and decided on inclusion or exclusion according to criteria described in the following section. Disagreements were solved by consensus.

Types of Studies Included

We included retrospective studies conducted in intensive care unit settings which used data obtained from EHRs, clinical databases generated from EHRs, or through manual chart abstractions. We excluded studies which presented exclusively outpatient or emergency department data.

Data Extraction

For every included study paired reviewers (MB, CD, JT, and JS) manually identified and extracted—using a purposefully built online form—all elements characterizing the study’s inclusion criteria, the interventions or exposures being studied (or the comparison group), and primary outcomes as defined by the original study authors following the Patient/Population, Intervention, Comparison, Outcome (PICO) framework [14]. Each attribute was then classified according to the clinical type (diagnosis, vital sign, laboratory result, medication, etc). When these elements contained a temporal dimension as defined by Boland et al [15], they were abstracted as temporal intervals or instants. For example, if the study included patients that underwent mechanical ventilation, because mechanical ventilation occurs during a period of time, such inclusion criteria would be abstracted as a mechanical ventilation interval; in the case of a single dose of antibiotics, that would be abstracted as a drug administration instant. Attributes that were not suitable to be represented as temporal attributes—such as sex, race—were represented as nontemporal patient attributes. A representation of the data extraction process can be seen in Figure 1.

Figure 1. Overview of the data extraction process.
View this figure

When possible, if an interval or instant was itself an abstraction of lower-level concepts, it was decomposed into its parts according to the description explicitly provided or cited by the authors. If there were no details in the paper, we used standard definitions, when available. For example, when a systemic inflammatory response syndrome [16] was used as an inclusion criterion, we abstracted its components as determined by systemic inflammatory response syndrome definition at the time of the study: body temperature, heart rate, respiratory rate, arterial CO2 pressure, and white blood cells. If standard definitions were not available, we did not decompose that interval and it was extracted as the authors described it. Clinical events that are stored as free-text format—whether because they are traditionally stored in this form or it is the only available format, such as radiology reports or surgical protocols—were not represented in the abstractions.

To minimize variability in the data extraction process, all researchers followed an initial training period. Researchers classified the identified elements—inclusion criteria, interventions or exposures, and outcomes—using the framework described above. Discrepancies on the concept extraction and temporal representations were resolved by group agreement. We performed descriptive statistics from the concept extractions and the temporal elements obtained.

The abstraction process was conducted iteratively and continued until the point of saturation. We predefined saturation as being met when including additional studies did not add any new types of temporal elements.

Finally, researchers systematically documented temporal and nontemporal relationships between the identified temporal elements. This allowed us to identify the temporal query design patterns present in the literature. Finally, we documented the required functionality for a novel temporal abstraction-based system to identify patient cohorts, interventions/exposures, and outcomes in large clinical databases.

Data Extraction

After iteratively extracting clinical concepts, the point of saturation—where no new types of temporal elements were identified—was reached after reviewing 28 primary studies. We obtained a total of 362 clinical entities, 48.6% (n=176) were inclusion criteria, 24.3% (n=88) were classified as interventions or exposures, and 27.0% (n=98) were outcomes. Abstracted entities were further classified into categories according to their clinical type, which are described, with examples, in Table 1. Therapeutic interventions (26.2%, 95/362), diagnostic tests (20.7%, 75/362), and vital signs (11.3%, 41/362) categories covered almost 60% of all entities.

Table 1. Categories, examples, and frequencies of identified clinical entities (N=362).
ClassificationExampleCount, n (%)
Therapeutic interventionDrugs or procedures: vancomycin, orotracheal intubation95 (26.2)
Laboratory/diagnostic testsSerum creatinine, hematocrit75 (20.7)
Vital signsBody temperature, respiratory rate, central venous pressure41 (11.3)
DiagnosisPneumonia, urinary infection35 (9.7)
Patient locationIntensive care unit hospitalization, patient transfer26 (7.2)
Clinical scoresAPACHE IIa, Cerebral Performance Category25 (6.9)
Nontemporal attributeSex, ethnicity17 (4.7)
DeathIn-hospital deaths, 30-day mortality15 (4.1)
Physical examination findingPupil diameter, abdominal pain11 (3.0)
Past medical historyHistory of trauma7 (1.9)
DispositionDischarge to home, institution, or other health center5 (1.4)
OtherAppropriate antibiotic usage10 (2.8)

aAPACHE II: Acute Physiology And Chronic Health Evaluation II.

Temporal Entities—Instants and Intervals

Of the 362 abstracted entities, 328 could be classified as clinical instants or intervals. Most entities could be abstracted as instants (54.1%, 196/362). This type of abstraction is used to represent a clinical event that does not have a duration but has a timestamp. For example, one inclusion criteria in this category was “the presence of arterial lactate > 2.5 mmol/L.” As much as 36.5% (132/362) of abstracted entities were of type interval. This type of abstraction is used to represent a clinical event that has a duration—defined by a start and end time—greater than 0. An example of a clinical interval is “noninvasive mechanical ventilation for at least 48 hours.”

Types of Clinical Intervals

Further analysis of clinical intervals showed that they can also be subdivided into 3 different categories:

  • Instant-based intervals: clinical intervals that are abstractions of identical instants. An example of this is hypothermia interval in which the interval is an abstraction of multiple instants of low body temperature measurements. Sometimes specific conditions have to be met to abstract this kind of interval: a time interval for a patient receiving more than 100 mL/hour of intravenous fluids. In other occasions, the instants were only used as categorical variables, regardless of the quantity: patient receiving normal saline infusion.
  • Bounded intervals: clinical intervals that are abstractions of specific instants defining their start and end times. An example of this is a hospitalization interval, where the start is defined by an admission instant and the end is defined by a discharge instant. Additional arithmetic operations may need to be applied to these intervals, for example, a clinical interval describing a hospitalization longer than 7 days.
  • Moving window intervals: clinical intervals where a specific condition needs to be met during a predefined window of time. An example of this was an oliguria interval, in which the condition oliguria (urinary output < 0.5 mL/kg/hour) has to be met during a 6-hour window. This denomination is consistent with previous descriptions [17].

Graphic examples of the 3 types of intervals are presented in Figure 2.

Figure 2. Three observed categories of clinical temporal intervals.
View this figure

Within-Interval Calculations

In a small subset of intervals, arithmetic calculations were needed to correctly abstract them. For example, calculating an interval of pulse pressure variation (PPV) within a defined range would require calculating PPV (%) = 100 × 2 ([PPmax – PPmin]/[PPmax + PPmin]) at each instant before executing the abstraction. Other examples of within-interval calculations included counting the number of instants occurring inside an abstracted interval. An example of this would be an outcome defined as the number of chest x-rays performed on each patient during his or her stay in the intensive care unit; the interval is of type bounded (admission/discharge from the intensive care unit) and we need to count the number of additional instants (chest x-rays) occurring within the interval.

Temporal and Atemporal Relations

We explored the temporal relations between instants and intervals and, as expected, all of them conformed to the temporal logic described by Allen [18]. Briefly, Allen described 13 possible temporal relations between a pair of intervals. Examples of these are the before, equal, and overlap temporal relations, among others. Graphic examples are presented in Figure 3.

Figure 3. Examples of temporal relations.
View this figure

In addition, some intervals were constructed by combinations of Boolean relations between intervals and instants. For example, to adequately represent a pediatric sepsis interval as defined by the International Pediatric Sepsis Consensus Conference [19] as required by the study authors, we required the Boolean relation AND between 6 different instants, and each one of them temporally related to an instant-based interval.

Some of the extracted entities did not have a temporal component and were denominated nontemporal patient attributes. Examples of these are age, race, and sex.

Finally, 5.5% (20/362) of the extracted concepts were not able to be represented using this proposed framework. For example, the outcome appropriate antimicrobial administration defined as whether the isolated bacteria were susceptible to the administered antibiotic implies a qualitative interpretation of a laboratory examination, which is out of scope of a temporal representation of clinical entities.

Nested Queries

One additional functionality that was particularly salient was the need to perform nested queries. In a nested query, a query uses the output of another query as its input. Observational studies frequently explore the effect of a specific exposure; this study design involves creating 2 patient cohorts that are identical except for the exposure. When the outcome is assessed in both cohorts, a nested query is the most natural way to satisfy this requirement:

SELECT (Outcome Phenotype) FROM (Exposure Cohort)

In this case Outcome Phenotype and Exposure Cohort are both themselves queries.

Design Patterns

The combination of temporal entities (instants and intervals), temporal relations, and nontemporal patient attributes can be used to describe the different observed patterns. A graphic description is presented in Figure 4.

Intervals can be temporally related to either instants or intervals. The same was observed for instants. We observed all 13 temporal relations described by Allen [18]. We call a pair of temporally related temporal entities (ie, Interval–Relation–Interval) a basic pattern. These basic patterns can, in turn, be related to other temporal entities or other temporal patterns. Those relations can be either temporal or through Boolean operators (AND, OR, Exclusive OR, NOT).

Intervals can use external variables as a condition to meet either before or after being abstracted. The first case would be the abstraction of an interval of reduced urinary output (<5 mL/kg/hour), in which each urinary output instant needs to be checked against the patient’s body weight (the external variable) before being added to the interval. An example for the second case could be total dose of prednisone less than 10 mg/kg. This interval is abstracted from individual instants of prednisone administration and after the interval is abstracted, it is checked against the patient’s body weight (the external variable). A final case was seen when an internal calculation—using information completely contained within the interval—was needed to be performed to generate the required attribute for the interval. An example of this would be an interval of a series of chest x-rays and, at the end, the number of x-rays would be calculated to create the interval total number of chest x-rays per week.

Figure 4. Examples of identified clinical temporal design patterns. ICU: intensive care unit.
View this figure

Main Findings

This study presents a systematic, literature-based assessment of design requirements to develop a temporal abstraction-based digital phenotyping tool. Such a tool would facilitate the conduction of retrospective clinical studies in critical care using routinely collected electronic clinical data through enabling a rich description of clinical phenotypes. Once validated, these temporally abstracted digital phenotypes should be able to correctly represent patient cohorts, clinical interventions or exposures, as well as relevant clinical outcomes. The iterative nature of this review, which was conducted until reaching information saturation, adds robustness to its findings.

The initial findings of this review are consistent with previous research describing the nature of temporal clinical entities, in the form of clinical instants and intervals [20], as well as temporal relationships between these entities [18]. Other temporal abstraction-based digital phenotyping systems have been described in the past [21,22]; however, there are no reports that their development has been informed by systematically reviewing observational studies. As a consequence, this study adds 3 additional functionalities that may facilitate the creation of digital phenotypes for observational research.

First, this review shows that 3 subtypes of clinical intervals—instant-based, bounded, and moving window—are necessary to adequately represent digital phenotypes. Second, in addition to these interval subtypes, there is a need to perform calculations both within a clinical interval and with data external to the interval being abstracted. The third component involves the need to allow for nested queries when building digital phenotypes for observational studies. Other findings of this systematic review confirm the need to query for temporal relations and Boolean relations as described by Mo et al [23] in their desiderata for digital phenotyping. Finally, it is essential to highlight the need to generate high-quality temporal metadata during routine clinical documentations because temporal queries are an essential component of digital phenotyping.


The main limitation of this review is its focus only on intensive care studies. We chose this setting given the temporal density of clinical data collected during critical care episodes. We cannot claim that these findings will be similar in other clinical domains; that statement would need to be explicitly verified in additional studies. A second limitation is the exclusion of inclusion criteria based on free text contained in clinical notes or reports. This was an explicit decision given our goal of designing a digital phenotyping system able to abstract higher-level concepts from structured data without relying on free text. We still need to demonstrate the feasibility of this approach [24].


This work was supported by DC’s CONICYT-FONDECYT (Chile) grant (no. 11130577).

Authors' Contributions

MB contributed with data analysis, manuscript write-up, and final document review. CD, JS, and JT contributed with data analysis and final document review. DC contributed with study conceptualization and design, data analysis, manuscript write-up, final document review, and decision to submit.

Conflicts of Interest

None declared.

  1. Schäferhoff M, Martinez S, Ogbuoji O, Sabin ML, Yamey G. Trends in global health financing. BMJ 2019 May 20;365:l2185 [FREE Full text] [CrossRef] [Medline]
  2. Westreich D, Edwards JK, Lesko CR, Cole SR, Stuart EA. Target Validity and the Hierarchy of Study Designs. Am J Epidemiol 2019 Feb 01;188(2):438-443 [FREE Full text] [CrossRef] [Medline]
  3. Tunis SR, Stryer DB, Clancy CM. Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy. JAMA 2003 Sep 24;290(12):1624-1632. [CrossRef] [Medline]
  4. Palabindala V, Pamarthy A, Jonnalagadda NR. Adoption of electronic health records and barriers. J Community Hosp Intern Med Perspect 2016;6(5):32643 [FREE Full text] [CrossRef] [Medline]
  5. Budrionis A, Bellika JG. The Learning Healthcare System: Where are we now? A systematic review. J Biomed Inform 2016 Dec;64:87-92. [CrossRef] [Medline]
  6. Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, et al. A Harmonized Data Quality Assessment Terminology and Framework for the Secondary Use of Electronic Health Record Data. EGEMS (Wash DC) 2016;4(1):1244 [FREE Full text] [CrossRef] [Medline]
  7. Capurro D, Yetisgen M, van Eaton E, Black R, Tarczy-Hornoch P. Availability of structured and unstructured clinical data for comparative effectiveness research and quality improvement: a multisite assessment. EGEMS (Wash DC) 2014;2(1):1079 [FREE Full text] [CrossRef] [Medline]
  8. Ross J, Tu S, Carini S, Sim I. Analysis of eligibility criteria complexity in clinical trials. Summit Transl Bioinform 2010 Mar 01;2010:46-50 [FREE Full text] [Medline]
  9. Overby CL, Weng C, Haerian K, Perotte A, Friedman C, Hripcsak G. Evaluation considerations for EHR-based phenotyping algorithms: A case study for drug-induced liver injury. AMIA Jt Summits Transl Sci Proc 2013;2013:130-134 [FREE Full text] [Medline]
  10. Shivade C, Raghavan P, Fosler-Lussier E, Embi PJ, Elhadad N, Johnson SB, et al. A review of approaches to identifying patient phenotype cohorts using electronic health records. J Am Med Inform Assoc 2014;21(2):221-230 [FREE Full text] [CrossRef] [Medline]
  11. Khan A, Ramsey K, Ballard C, Armstrong E, Burchill LJ, Menashe V, et al. Limited Accuracy of Administrative Data for the Identification and Classification of Adult Congenital Heart Disease. J Am Heart Assoc 2018 Jan 12;7(2):e007378 [FREE Full text] [CrossRef] [Medline]
  12. Weng C, Tu SW, Sim I, Richesson R. Formal representation of eligibility criteria: a literature review. J Biomed Inform 2010 Jun;43(3):451-467 [FREE Full text] [CrossRef] [Medline]
  13. Hruby GW, Matsoukas K, Cimino JJ, Weng C. Facilitating biomedical researchers' interrogation of electronic health record data: Ideas from outside of biomedical informatics. J Biomed Inform 2016 Apr;60:376-384 [FREE Full text] [CrossRef] [Medline]
  14. Huang X, Lin J, Demner-Fushman D. Evaluation of PICO as a knowledge representation for clinical questions. AMIA Annu Symp Proc 2006:359-363 [FREE Full text] [Medline]
  15. Boland MR, Tu SW, Carini S, Sim I, Weng C. EliXR-TIME: A Temporal Knowledge Representation for Clinical Research Eligibility Criteria. AMIA Jt Summits Transl Sci Proc 2012;2012:71-80 [FREE Full text] [Medline]
  16. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest 1992 Jun;101(6):1644-1655. [CrossRef] [Medline]
  17. Gall W, Duftschmid G, Dorda W. Moving time window aggregates over patient histories. Int J Med Inform 2001 Oct;63(3):133-145. [CrossRef] [Medline]
  18. Allen J. Maintaining knowledge about temporal intervals. Commun. ACM 1983 Nov;26(11):832-843 [FREE Full text] [CrossRef]
  19. Goldstein B, Giroir B, Randolph A, International Consensus Conference on Pediatric Sepsis. International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics. Pediatr Crit Care Med 2005 Jan;6(1):2-8. [CrossRef] [Medline]
  20. Shahar Y. A framework for knowledge-based temporal abstraction. Artificial Intelligence 1997 Feb;90(1-2):79-133 [FREE Full text] [CrossRef]
  21. Post AR, Kurc T, Willard R, Rathod H, Mansour M, Pai AK, et al. Temporal abstraction-based clinical phenotyping with Eureka!. AMIA Annu Symp Proc 2013;2013:1160-1169 [FREE Full text] [Medline]
  22. Mate S, Bürkle T, Kapsner LA, Toddenroth D, Kampf MO, Sedlmayr M, et al. A method for the graphical modeling of relative temporal constraints. J Biomed Inform 2019 Dec;100:103314. [CrossRef] [Medline]
  23. Mo H, Thompson WK, Rasmussen LV, Pacheco JA, Jiang G, Kiefer R, et al. Desiderata for computable representations of electronic health records-driven phenotype algorithms. J Am Med Inform Assoc 2015 Nov;22(6):1220-1230 [FREE Full text] [CrossRef] [Medline]
  24. Hernandez-Boussard T, Monda KL, Crespo BC, Riskin D. Real world evidence in cardiovascular medicine: ensuring data validity in electronic health record-based studies. J Am Med Inform Assoc 2019 Nov 01;26(11):1189-1194 [FREE Full text] [CrossRef] [Medline]

APACHE II: Acute Physiology And Chronic Health Evaluation II
EHR: electronic health record
PICO: Patient/Population, Intervention, Comparison, Outcome
PPV: pulse pressure variation

Edited by G Eysenbach; submitted 09.07.20; peer-reviewed by L Jordan, G Jiang; comments to author 17.08.20; revised version received 30.08.20; accepted 28.10.20; published 24.11.20


©Daniel Capurro, Mario Barbe, Claudio Daza, Josefa Santa Maria, Javier Trincado. Originally published in JMIR Medical Informatics (, 24.11.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.