All papers included in the analysis were passed through the data extraction form and resulted in an initial comprehensive matrix of results. Of 74 questions that were initially probed, results that yielded over 75% not reported, or not applicable across all papers analyzed were removed from our analysis. Then 39 variables were selected for inclusion in the initial matrix. Phase 1 results are summarized in the comprehensive matrix of design properties (Table 1 and Figure 1) and phase 2 results are summarized in the Comprehensive Matrix of Study Results (Table 2 and Table 3). The comprehensive matrix of design properties presents 10 variables which are divided between 2 tables. Variables appearing in (Table 1) are “Target Users”, “Year”, “Clinical Context”, “Number of Variables”, and “Display Type”.

“Target Users” relates to the clinical specialty, and “Year” is the approximate date the prototype was tested. Due to the difference between the dates of publication and evaluation, this value was approximated based on the date of submission of the article. “Clinical Context” conveys the copresence of contextual clinical information, and “Number of Variables” refers to the total number of physiological or clinical variables that were visible in a single screen. “Display Type” lists the types of graphics utilized by the paper belonging to one of: tabular, object, or metaphoric displays. “Color(s) Used” identifies the hue where available. “Pre-attentive Processing” lists particular visual variables that were used in the visual representation such as: shape, size, and dimension. “Gestalts” refers to the designer’s use of grouping laws identified by Gestalt’s laws of perception: the use of proximity, similarity, closure, symmetry, and continuity as a means of discerning visual objects presented in the display [5]. Finally “Interactive Controls” refers to the ability for the display to support direct manipulation of one or more properties and “Iterative Design” identifies displays that were built using user-centered design approaches that include users into key decision making processes prior to the development of the display.

A second matrix, titled the Comprehensive Matrix of Study Results presents additional 11 variables that were identified in papers which presented study results. Table 2 lists “Setting,” “Study Type,” “Results Reported,” “Realism,” “Cognitive Workload,” “Historic Trends,” “Visual Encoding for Temporal Trajectory,” “Visual Encoding for Duration,” “Visual Encoding for Frequency”. Table 3 lists “Counter-balanced for Display or Scenario,” “Were Scenarios Clinically Relevant,” and “Function Supporting Case-controlled Analysis.” “Setting” describes the location where the study was physically conducted; for instance, the lab, clinic, or public areas. “Study Type” identifies research method used to validate the display. “Results Reported” summarizes key findings from the study, and “Realism” presents the latency of the display as well as the ability of the display to mimic real-world dynamism. “Cognitive Workload” reports on findings indicating reduced or increased workload and “Historic Trends” identifies displays that present historical trends that are greater than 5 minutes. The variables beginning with visual encodings for temporal trajectory, duration, and frequency identify particular techniques used by the displays to represent trends, duration of events, and frequency of events. The counter-balanced variable identifies methodologies that used strategies to minimize learning effects during the experiment. Finally, the clinical scenario variable lists the displays that were evaluated using real-world clinical scenarios. These tables, along with descriptions of the results are presented in the next section.

The goal of the comprehensive matrix of design properties is to present design decisions that were followed to develop prototypes across all 39 papers analyzed. Visual representations were found across mainly anesthesiology (n=17), critical care (n=20), and in some multi-discipline (n=2) environments. Only one display was developed as a tool for intensive nurses [42]. Multi-discipline environments consist of 2 or more specialities, such as integrated in-patient and out-patient systems. Visual displays started to become actively contributed from the early 1990s, then increasing every 10 years, 1984 (n=1), 1985-1994 (n=8), 1995-2004 (n=13), and 2005-2016 (n=17). Integrated clinical data was also found across some displays (n=16), while a greater number of displays were devoted to the display of physiological waveforms (n=24). Number of variables presented in a single screen was wide-ranging; most displays contained greater than 20 variables per screen (n=15), followed by 11-20 variables (n=12), while 7 displays contained between 0-4 variables.

Reviewed visual representations included a mix of display formats, such as tabular (TB), object-based (OB), waveform (WF), and metaphoric (MT). Standalone MT representations were most commonly seen (n=12), followed by standalone WF (n=6). With respect to combinatory displays, TB appeared with WF (n=6) most frequently, WF with MT (n=6), and followed by WF with an OB (n=4). When identifying the display type most frequently paired in a combinatory display, WF (n=12) appeared most often, followed by TB (n=7) and OB (n=6) displays. Overall, across all identified papers including those where multiple representatives were presented, metaphoric displays were the most popular (n=22), followed by waveform displays (n=20), and object displays (n=10).

Visual representations utilized at least 2 of the primary colors, red, blue or green (n=21), while yellow (n=11) and turquoise (n=4) were also popular options. Three papers utilized discrete color encoding, 2 papers [25,43] mentioned the source of their color coding. A number of papers did not specify the type of color that was used (n=10). Pre-attentive processing of items were commonly exploited through manipulating visual variables such as color (n=24) and size (n=12), followed by dimension (n=7), and shape (n=5).

Visual representations also exploited some aspect of Gestalt’s law of groupings, such as continuity (n=18) with waveform displays, closure (n=17) when identifying boundaries, symmetry (n=14) with visual metaphors and object-based displays, and proximity (n=7) to aid in higher level detection of abnormal events. The most popular interaction method that was supported was selection (n=13). Selection allows the user to select visual objects directly to reveal greater details. This was followed by interactive filtering (n=7) to select partial ranges such as short durations of time. Finally, in many cases designs were proposed without following user-centered design approaches (n=28). In total, 10 papers reported using user-centered design processes, while 4 papers described a structured approach used in developing the proposed visual design [5,31,43,44].

The Comprehensive Matrix of Study Design (Figure 2) presents the results that were reported by authors for any evaluation or experiment. While the search strategy yielded 39 full papers that were identified for analysis, only 29 of these papers contained primary study results from a case study, evaluation, or human experiment, and employed 1 of naïve, novice, or expert participants in the evaluation method. Naïve participants were generally undergraduate students with little or no prior clinical knowledge. Novice participants ranged from undergraduate computer science or nursing students to newly graduated clinical staff. Expert participants had at least 10 years of experience.

The number of participants exposed to test conditions highly varied; however, the majority of studies employed at least 15 participants. Six studies used a sample size greater than 20 to test for detection, diagnostic, and treatment accuracy, with the minimum and maximum being 4 and 32 participants, respectively. Most displays integrated these systems in a single display using live or static representations (n=15), while displays that were presented as case studies (in situ) were connected to central monitoring systems. Some displays supported views of clinical information that integrated data from other clinical and laboratory systems (n=15) [45]. Most prototypes that were evaluated used more than one data stream, with the exception of the studies that contained low-frequency updates (n=9). Most evaluations or experiments utilized more than one condition to test each display; however, a few did not have any scenarios or patient conditions (n=9). A large number of studies also did not utilize data from more than one patient-source (n=26).

Most of the studies were conducted in laboratory environments (n=24), followed by evaluations or experiments in the intensive care unit (n=12). Some studies were evaluated over multiple specialities (n=2). A majority of studies used some form of experimentation to validate their designs (n=21), although the specific method of experimentation was not always explicitly mentioned. Evaluations involved clinicians and mixed qualitative and quantitative methods were used to report results (n=8). Applications were primarily qualitative in nature, often depicting results through anecdotes (n=7). The remaining studies were design papers that investigated novel visual representations without involving prototypes. Of the papers that reported results (n=31), most reported positive findings (n=27), but in some cases negative results were also reported (n=4). A between-group experimental study yielded site-dependent results that were skewed towards the site that produced the visual representation. For evaluations or experiments the source of data to support realism was spread across live simulations (n=19), live patient-origin data (n=9), or static patient-generated data (n=11). Most studies did not test for cognitive overload using ad hoc methods or traditional workload score metrics such as the NASA Task Load Index (NASA-TLX) (n=30). Where cognitive workload was reported (n=7), most were reported to have reduced cognitive overload (n=5), while others reported no change or mixed results (n=3).

Long-term historical values, specifically ranges exceeding 5 minutes of monitoring were not included in majority of the displays (n=28). Tri-event parameters, namely, trajectory, frequency, and duration were seldom supported by visual representations, where these parameters were identified, trajectory was most frequently found (n=27). Temporal trajectory was encoded using curves (n=25) such as in a line graph, or glyphs (n=9). In terms of duration, the second tri-event parameter was seen across 10 displays, of which, glyphs (n=6), text (n=4) or curves (n=2) representations were utilized. Frequency, the last tri-event parameter was also seen in some visual representations encoded by glyph (n=5) or text (n=4) where supported. Where displays were validated through experimentation, both the display and scenarios were more often counterbalanced (n=12), while some experiments counterbalanced only the scenario (n=6) and others only the display (n=4). Scenarios were utilized across many studies utilizing experimentation or evaluation methodologies (n=22) and most were clinically relevant problems (n=21). Finally, only one of the evaluated visual representations supported the ability to perform analysis across multiple patients.

A total of 19 novel visual representations were identified from analysis of the literature. Novel displays were seen across 4 main groups, including tabular, waveform (graph-based), object, and metaphors. The latter 2 are aggregated together as ecologic displays.

The early- 1990’s saw growing interest in converting large volumes of paper patient charts to “virtual” records [24,46-49]. Initial representations adopted by these virtual patient records were largely tabular and text-dominant, and sometimes contributed negatively to information overload [48]. Figure 2 [50] illustrates an example of a traditional virtual patient chart that mimics a traditional paper flow chart. This review identified 14 tabular representations published from 1952 to 1997. Those systems provide a direct manipulation using the traditional desktop-oriented, Windows-Icon-Mouse-Pointer (WIMP) interaction paradigm. Additional levels of interactions, such as multiple mouse clicks, are required to access unique views of patient data. Large number of these displays are often duplicated to a physical copy, in part due to the simplicity and ease of reading paper charts [51].

The review identified 9 out of 39 studies that used some form of live physiologic streams from real patients to display largely identical waveform representations. It was also noted that much of these waveform displays were integrated with other tabular and text representations. Five papers that presented waveform displays also supported interactive capabilities, including the ability to select regions of interest, filter based on patients, and generate screen captures [6,8,25,43,52]. Stylianides and colleagues (2011) present an engine for producing waveform graphics [23]; however, their system serves the purpose of animating historic physiologic data streams. CareCruiser [25], supports the interactive exploration of treatment plans using physiologic data. However, that system was not evaluated using more than one clinical user. PhysioEx [43] was evaluated using an expert evaluation methodology employing 5 domain experts, and was shown to further enhance that interactive analytic workflow by providing coordinated analysis of temporal data streams; however, using waveform displays only to guide the user with additional context.

Despite their ability to communicate acute time-sensitive events [20], waveform representations have numerous limitations [4,31,53]. One prime disadvantage of waveform displays is the potential to negatively impact cognitive load, that is, they require humans to monitor and consume large numbers of data points as they are produced to derive trends and higher level knowledge [7,8,22]. These waveforms display can convey several features in one frame; therefore, easily disturb limited resources of the working memory capabilities [54]. The challenge of managing large volumes of data have been extensively studied in several domains, such as information overload [55], visual data mining [56], and addressing cognitive challenges related to interruptions, task performance, and decision making [55,57-59].

Integrated methods of representing critical physiological information have been actively studied to reduce the internal mental processing requirement [20,22,32,60-62]. These integrated displays use a combination of text [33,34], graphic [3,4,63], and waveform [64,65] representations to summarize low-level information. Figure 3 [6] illustrates an example of an integrated display. Three such integrated displays were identified in the review [6,8,25]. These displays support clinicians to interactively select regions of interest while monitoring other forms of slow-changing clinical data. However, only one display allows the clinician to compare against a cohort [25]. Other studies, seeking alternatives to the waveform visual encoding, propose novel and ecological methods to improve knowledge discovery and minimize cognitive overload.

Physiological displays can be designed and developed using 3 consumption efficacy metrics derived for temporal and dynamic data streams [43]. These metrics are termed tri-event temporal parameters namely, trajectory, frequency, and duration of salient events.

Among the tri-event parameters, trajectory was found to be the most popular, with 32 of 39 studies incorporating some form of trajectory information. However, longitudinal trajectory was found in only 9 studies, and was rare among displays that were found in anesthesiology but more common in critical care. Displays that incorporated an aspect of the tri-event temporal parameters exclusively adopted trajectory. Nine visual representations were found to have included the duration and frequency metrics. Most of the representations that included duration and frequency used glyphs (n=6) or text (n=5) to communicate episodic information. For instance, PhysioEx [43] uses the river metaphor [72] to illustrate frequency of adverse physiologic events that were analyzed by a real-time algorithm (bottom left, in Figure 10). Text also remains a popular method for communicating discrete events. Law and colleagues found text to be superior to waveform and numeric displays when communicating clinical episodes, even while clinicians reported a preference for graphical displays [73]. Where multiple views were presented, only one representation utilized interactive coordination between independent views [6].

Visual representations of physiological data have been attempted several times as witnessed by the sheer size of prior work discussed in this paper. Many have shown their potential to improve clinical care, and while largely positive results have been released, there are still concerns as to the efficacy of both in reproducibility as well as translatability to the unit. In particular, methods to identify the accuracy of actions post-treatment to the display remain as concern and open areas for further exploration.

Few clinical visualization papers studied associations of the treatment condition to the accuracy or accrued insight by the user. It was also seen that most studies included detailed study of the time to diagnosis and its accuracy; however, many of these studies included highly controlled scenarios with highly visible graphical distortions. Additionally, few studies used real patient data to evaluate their prototypes. Hence, the frequency of events with clear and distinctive graphical patterns existing across real patient data remains untested. Detection was also another area where studies frequently report positive findings; however, in many cases these differences were marginal and found in narrow statistical ranges. It has yet to be proven whether these statistical significances are relevant in the clinical domain. Exact mechanisms inducing positive effect have yet to be studied within the prototypes studied [63,74].

Visual representations show promise; however, they are plagued with user-preference and interaction challenges. Results spanning two decades continue to show positive influence of graphical representations when they are used in simulated studies [4]. However, many of these studies have not used standardized metrics to test distinct controlled variables, or provide evidence of precisely which features of the graphical displays afford greater comprehension to the consumer. Questions still remain as to its efficacy in clinical practice, where, the availability of all data required by the representations may be limited. There is also the limitation of graphical representation failing to maintain interpretable coherence, when provided incorrect data [2].

Some studies have also demonstrated user involvement as an important factor which may have influenced results, in the design and development of the clinical system [45]. Future studies should focus on clinical validation as a means to identify real-life relevance. Clinical experiments are difficult in lieu of several considerations and their limitations. However, one study by Wachter et al [9], demonstrates that observational studies, although somewhat intrusive, may produce some significant qualitative results. These studies need to be expanded, and clinical trials must ultimately demonstrate their efficacy. Cognitive errors also require additional research effort, specifically by including evaluation methodologies such as the NASA-TLX score to allow end-users to self-report perceived workloads.

Only 7 visualizations were identified to have had some element of interactive selection and filtering functions to support basic analysis tasks. While only one display was identified to support analysis across cohort populations. The general absence of analysis functionalities is an opportunity for enhancing physiologic visualizations. Physiologic data represents a unique subset, due to the dynamic and streaming nature of the data. Application of visual analysis techniques may support novel uses of physiologic visualizations, such as supporting human-driven hypothesis generation tasks.

Finally, research in visual representations should include tri-event parameters as important design considerations to produce designs that communicate episodic information. PhysioEx was seen to incorporate all 3 parameters; however, it was limited to one view per patient [43]. These visual representations can then be used to better assess the influence of tri-event parameters on higher level workflows as well as in the progression of clinical conditions.