IntroductionType 2 diabetes (T2D) is a common chronic disease affecting more than 11% of the US population [1]. Given the rapidly increasing burden of T2D on health care services and resources, health systems have devised strategies to address such demands by identifying and managing T2D patients across their populations [1]. However, despite clinical guidelines to identify T2D at the point of care, identifying T2D patients in large clinical repositories is still a challenge [2]. Ambiguity of algorithms to identify T2D patients in complex data sets and data quality issues in routine clinical data sources such as electronic health records (EHRs) are still hindering the development of generalizable approaches to identify T2D patients across different health systems.
Over the last two decades, several approaches have been developed for health care professionals to identify patients with T2D in large clinical data repositories. Multiple T2D phenotype definitions (also known as computational algorithms) are available that define characteristics to identify T2D populations. Some examples of T2D definitions include the Surveillance, Prevention, and Management of Diabetes Mellitus (SUPREME-DM) [3], the Centers for Medicare and Medicaid Services (CMS) Chronic Conditions Data Warehouse (CCW) [4], the Electronic Medical Records and Genomics (eMERGE) Northwestern Group [5], and the Durham Diabetes Coalition (DDC) [6] phenotypes. Each phenotype definition has its own set of inclusion and exclusion criteria using different data types. Data types may include International Statistical Classification of Diseases, Tenth Edition (ICD-10) diagnosis codes, RxNorm medication codes, and Logical Observation Identifiers Names and Codes (LOINC) laboratory codes, with sequences and frequencies of occurrences of diagnosis, medication and laboratory codes; defined time periods; and care pathways. However, despite these detailed definitions, it is unclear which of the existing T2D phenotypes are prone to inherit data quality issues from clinical data sources such as EHRs. The uncertainty of T2D phenotypes’ performance for identifying populations with T2D using real-world data has led to the lack of a universally agreed-upon T2D phenotype.
The concept of data quality in health care varies based on the problems and functional needs of end users. Health care providers, data scientists, or policy makers may differ in their approach to data quality and its significance in practice [7]. Typically, health care data such as captured in EHRs are collected for continuity of clinical care, coding, and billing purposes, and not necessarily to answer specific research questions. Thus, the quality of data collected in EHRs may be sufficient for clinical purposes but may not meet the needs of a researcher or population health intervention. For example, a clinician may identify a patient with T2D at the point of care despite having incomplete data; however, a T2D phenotyping algorithm may miss the same patient in a large EHR data warehouse due to the underlying data quality issues; hence, the patient may inadvertently be excluded from a research study or population health intervention.
Various data quality frameworks have been proposed to measure the quality of health care data. Completeness, accuracy, and timeliness are a few key data quality characteristics that are used across several frameworks [7-9]. The assessment for completeness defines how complete the data are, what the missing elements are, and how usable the data are in their “as is” format. Accuracy is the correctness and consistency of the data elements [8], and timeliness is how recent or current the data are for research and analysis. The assessment of data quality in health care is crucial with the continuous and prominent use of EHRs; however, despite the increasing use of EHR data to identify patients with T2D, the effect of varying levels of key data quality characteristics (ie, completeness, accuracy, and timeliness) on T2D phenotypes is still unknown.
The ongoing challenge of understanding the effect of key data quality issues on T2D phenotypes is further exacerbated by the fact that T2D phenotypes use multiple data types, such as diagnosis codes, medications, and laboratory results. Additionally, given the variability of key data quality issues of these data types across EHRs [10,11], measuring the effect of key existing data quality issues on T2D phenotypes in one EHR may not translate into generalizable findings. For example, one provider’s EHR may suffer from incompleteness of diagnosis codes, while another provider’s EHR may be affected by inaccurate medication data. Thus, to measure the effect of an EHR’s data quality on T2D phenotypes, varying (simulated) levels of key data quality characteristics across all data types (ie, diagnosis, medication, and laboratory) used by T2D phenotypes should be studied. These simulated levels of key data quality issues will in turn help providers to compare their EHR data quality issues with the simulated levels, contrast the potential impact of such data quality issues on identifying T2D patients using various phenotypes, and eventually select the most suitable T2D phenotype for their EHR data.
Currently, evidence is lacking on the effect of data quality issues (eg, completeness, accuracy, and timeliness) and the identification of T2D populations in large clinical data sources such as EHRs. This gap in evidence is further amplified given the variations in characteristics of published T2D phenotypes and potential discrepancies in underlying data types (ie, diagnosis, medication, and laboratory) in EHRs. To address these gaps, our study aimed to assess the impact of varying (simulated) levels of data quality issues across the different data types used by T2D phenotypes. Our study findings can inform health care providers and other stakeholders to select T2D phenotype algorithms that best match their underlying EHR data quality issues.
MethodsData SourceOur cross-sectional study used retrospective EHR data from the Johns Hopkins Medical Institute (JHMI) data warehouse over a 3-year period from 2017 to 2019. Clinical data included T2D primary diagnostic data (ICD-10), laboratory data (LOINC), and medication data (RxNorm). The demographic data included age, sex, race, ethnicity, and the patients’ residential state.
Study PopulationOur overall study population included approximately 208,000 patients in age groups of 18 to 90 years. This population denominator was identified and extracted in a prior study focusing on T2D patients and funded by the US Food and Drug Administration (5U01FD005942-05). This population denominator, also known as the raw data cut, was identified by the most inclusive query of the JHMI’s EHR data warehouse, which also included patients with mentions of diabetes in their clinical notes. Each of the identified T2D phenotypes was applied to the overall study population. At total of approximately 164,000 patients were included in at least one of the T2D phenotypes assessed.
Considering the size of the EHR data set, and given the impracticality of reviewing the individual records of almost a quarter of a million patients, no gold standard population was identified for this research. Indeed, the aim of this research study was not to assess the accuracy of the common T2D phenotypes; instead, we aimed to measure the performance of T2D phenotypes given the underlying data quality issues in EHR data repositories.
T2D Phenotype DefinitionsOur assessment included 4 published T2D phenotype definitions, from the CCW, DDC, SUPREME-DM, and eMERGE, as well as 1 unpublished definition (Johns Hopkins University; JHU). Since not all T2D phenotype definitions were defined with the most current ICD codes, we converted the DDC, SUPREME-DM, and eMERGE diagnostic phenotypes from ICD-9 to ICD-10 [12]. Additionally, not all phenotype definitions using medications had the list of specific RxNorm codes (eg, the DDC and SUPREME-DM phenotypes included only the names of the T2D medication and not the codes). We identified the RxNorm codes for each medication using the National Library of Medicine’s RxNav tool [13]. For the purposes of this study, we selected only those RxNorm codes that were associated with the primary ingredient or ingredients of the medication name in the T2D phenotype definition.
Of the 4 published T2D phenotypes, CCW only included diagnosis codes with a reference period of 2 years in the definition. The DDC, SUPREME-DM, and eMERGE phenotypes included a series of detailed care pathways with diagnosis, medication, and laboratory codes and results within specified time periods. However, the unpublished definition from JHU did not have pathways and was the most inclusive definition, as it included all data types (diagnosis, medication, and laboratory) with wide eligibility criteria to identify patients with T2D.
Factors Affecting T2D PhenotypingThe data included demographics, T2D-related data types (ie, diagnosis, medication, and laboratory), and the Charlson Comorbidity Index [14]. The demographic data included age, sex (male, female, and other), race (White, Black, Asian, and other), ethnicity (Hispanic/Latino or non-Hispanic/Latino) and patient location (state of Maryland or other locations). The “other” category for sex, race, and ethnicity was primarily composed of missing data entries. Since the JHMI is in Baltimore, Maryland, the majority (169,215/207,813, 81.4%) of our study population was from Maryland and the remainder (38,598/207,813, 18.6%) was from the surrounding states. The outcome measures included the extent of overlap in identifying patients with T2D and the degree of robustness against data quality issues across phenotypes.
Statistical AnalysisWe performed descriptive data analyses across the 5 different phenotypes to identify EHR populations with T2D. We used the χ2 test for categorical variables and ANOVA for continuous variables. Our analysis included distribution and overlap of the population of interest by diagnosis, medication, and laboratory data types across each of the 5 T2D phenotypes. We introduced methods that simulated or induced data incompleteness, inaccuracy, and lack of timeliness (eg, date shifting) to assess the robustness of each phenotype in capturing the populations of interest. We created unique analytical functions for each data quality issue considering the data types that were applicable across all T2D phenotypes.
To simulate or induce data incompleteness, our procedure randomly dropped codes at 10% increments, from 0% to 100%, for diagnosis, medication, and laboratory codes within each of the T2D phenotypes. Incompleteness was simulated up to 100% as no thresholds of data missingness were known to affect the performance of T2D phenotypes beforehand. Incompleteness was induced for diagnosis, medication, or laboratory codes independently of the other 2 data types. For each increment of incompleteness, T2D phenotypes were reapplied to identify a new (and logically smaller) cohort of patients.
To gauge the impact of inconsistency and inaccuracy in our denominator population for each T2D phenotype, the diagnosis and medication codes were replaced at random at increments of 10% (the same as for incompleteness) from 0% to 100% with another code of the same data type, including T2D and non-T2D codes. We included T2D codes to illustrate expected data quality issues that may impact the phenotypes’ performance. For example, 10% of instances of the ICD-10 diagnosis code E08 (ie, diabetes mellitus due to underlying condition) were replaced at random with the ICD-10 code E09 (ie, drug- or chemical-induced diabetes mellitus—endocrine, nutritional, or metabolic disease). For laboratory codes, the laboratory values were induced with a 2% incremental change from −100% to 10% in laboratory results. For example, our procedure for simulated or induced laboratory values yielded results of 5.6% to 5.8% for hemoglobin A1c (LOINC code 55454‐3). Additionally, we simulated or induced inaccuracy in units of laboratory data results. We did this by intentionally converting reported laboratory units from US standards to UK standards; in particular, blood glucose level units in mg/dl were converted to mmol/L, and hemoglobin A1c level from percentage to mmol/mol [15].
For timeliness, we simulated or induced date shifts at increments of 30 to 365 days for all phenotypes. For example, our procedure induced a forward shift on December 1, 2019, by 30 days, shifting the date to December 31, 2019, and so on. Lastly, we also induced compounded data quality issues (incompleteness, inaccuracy, and lack of timeliness) to understand each phenotype’s resistance to changes that occurred across diagnosis, medication, and laboratory codes simultaneously. Induced compounded incompleteness would mean dropping diagnosis, medication, and laboratory codes randomly at increments of 10% up to 100%. Induced compounded inaccuracy would mean replacing diagnosis and medication codes randomly at increments of 10% up to 100% (laboratory codes were not replaced; their values were manipulated instead). Induced compounded lack of timeliness would mean inducing date shifts across diagnosis, medication, and laboratory codes at increments of 30 to 365 days.
SQL queries were written for data extraction. All visualizations and statistical analyses were conducted using R (version 4.2.0; R Foundation for Statistical Computing) [16]. The overall findings are showcased as descriptive data tables, Venn diagrams, and line charts depicting the effect of simulated or induced data quality issues on identifying T2D populations using the common T2D phenotype definitions.
Ethical ConsiderationsThis study was reviewed and approved by the IRB committee of the Johns Hopkins School of Public Health (00014440). The deidentified population denominator used in this study is from a prior study funded by the US Food and Drug Administration (5U01FD005942-05).
DiscussionT2D is a common chronic disease with no known universally defined phenotype definition. This is further complicated by lack of assessments that are necessary to understand the quality of data in real-world health care settings against common phenotypes. In our study, we investigated how data quality issues of completeness, accuracy, and timeliness may impact the implementation of existing T2D phenotypes using different data types of diagnosis, medication, and laboratory codes extracted from common clinical data sources such as EHRs. Each data quality issue and phenotype definition may present unique implications at an unpredictable scale in any given health care setting. Collectively, these variables can pose challenges in identifying an eligible denominator population and implementing study population enrollment strategies for epidemiological studies, population health and management, disease surveillance, financial operations, and policies for populations with T2D.
Although each phenotype has distinct characteristics, our results showed considerable overlap in the population identified with T2D across the phenotype definitions (Figure 1 and Table 2). Given the uniqueness of the CCW phenotype, which only includes diagnosis codes, the CCW phenotype identified approximately 38.4% (79,967/207,813) of the overall study population with T2D, whereas the DDC phenotype, which was inclusive of all data types, identified approximately 67.2% (139,832/207,813) of the total population with T2D. This was similar to the 66.9% (139,231/207,813) of the total population identified by the Hopkins phenotype. Additionally, we observed that the T2D phenotype definitions of SUPREME-DM and eMERGE had similar characteristics and therefore resulted in the identification of approximately 43.2% (89,772/207,813) and 37.5% (77,977/207,813) of the overall study population, respectively. Moreover, when analyzing the distribution of population by data types, the identified populations were significantly different, with no noticeable patterns.
This study revealed significant findings resulting from simulated or induced data quality issues on the overall study population using EHR data across different T2D phenotype definitions. Induced incompleteness of diagnostic data showed the least impact on the DDC phenotype, however, identifying only approximately 60% of the population with T2D at 100% diagnostic incompleteness. The CCW phenotype was the most impacted, as 100% of the diagnosis codes were missing with 100% induced incompleteness given the characteristics of its phenotype definition. With more incomplete data, the uniqueness of the population identified by each phenotype may also shift, thereby showing a decrease in overlap of the population with T2D across all phenotypes. Although trends displayed by phenotype definitions of Hopkins and SUPREME-DM may seem similar, the quantification of the slightest differences in trends can have significant population health implications, resulting in compromised financial and logistical (eg, staffing needs) outcomes. Thus, it is important to understand the resistance of data quality issues across phenotypes given different data types.
The results for induced inaccuracy were similar to those for completeness, but data quality issues for timeliness showed a different trend. The phenotypes of eMERGE, CCW, and SUPREME-DM showed significant decreases in the number of identified patients when the majority of the diagnosis codes were replaced. However, induced timeliness continued to capture significant numbers of patients for all phenotypes, at well above 85%. While the impact of induced inaccuracies of laboratory values (in the negative direction) was mitigated by other pathways of the phenotypes, overall, completeness and accuracy showed a much larger impact on the identification of the population with T2D. The trends for the data quality issues of completeness and accuracy paint a similar picture, which may translate into considerations for population health interventions. The effect of data quality on identifying T2D populations using commonly available T2D phenotypes can affect a variety of interventions and outcomes, such as clinical research, financial analysis, staffing needs, and logistical issues, to name a few.
Lastly, induced compounded (diagnosis, medication, and laboratory) data quality issues showed different trends compared to induced single component (diagnosis, medication, or laboratory) data quality issues. Overall, all phenotypes captured fewer patients when there were compounded data quality issues compared to single-component data quality issues. The CCW phenotype was the most resistant phenotype across compounded incompleteness, inaccuracy, and lack of timeliness. The CCW phenotype’s robustness to compounded incompleteness owes to its reliance solely on diagnosis codes; other phenotypes with medication and laboratory components faced steeper drops in patients captured when medication and laboratory codes were both incomplete. When it came to induced compounded inaccuracy, the CCW phenotype demonstrated the greatest resistance until 80% of both diagnosis and medication codes were replaced, at which point the Hopkins phenotype became the most robust (Figure S19 in Multimedia Appendix 1). Similarly, compounded lack of timeliness became a bigger problem for phenotypes that rely on time interval for not only diagnosis, but also medication and laboratory codes. Simulating compounded data quality issues has implications for evaluating the fit of phenotype definitions for various data sources and availabilities of multiple data elements. For instance, if a particular data source only has reliable diagnosis codes, then the CCW phenotype would be the most robust. However, it may come with more false positives due to its sole reliance on diagnosis codes.
Our study highlights the importance of understanding the effects of data quality issues on phenotypes, particularly for common diseases such as T2D. The results from our study are novel and can inform how to better identify denominators for a given purpose, which may be beneficial for both research and operational decisions. Although there is no gold standard data set to compare and analyze baseline thresholds of impact on phenotypes, the results of this study can be used by any researcher using real-world data. Additionally, our study describes methods for assessment of multiple data quality issues that can be applied simultaneously in any clinical and health care research setting.
Our study has some limitations. First, the results discussed are based on the EHR data of JHMI over a period of 3 years; hence, generalizability of the findings may be limited to populations like that of this study population. That said, it may also be not applicable to populations of much smaller sample sizes. Second, our study results should be tested against the epidemiological trends and spread of T2D over time. Third, our study did not measure the embedded issues of data quality problems (ie, only simulated or induced data quality issues); understanding and resolving these issues at the beginning of any study can be vital. Fourth, we did not study other data quality dimensions such as concordance and provenance, the assessment of which can be important. Fifth, all T2D phenotypes used in this study relied on diagnosis, medication, and laboratory codes to identify T2D patients; however, some patients may only have clinically diagnosed information in physician notes (ie, unstructured data or free text) that will be missed using structured diagnosis, medication, or laboratory codes. As a result, there may be some false negatives that may have been excluded from our overall study population despite incorporating the most inclusive criteria for the raw data cut. Sixth, we explored compounded data incompleteness, inaccuracy, and timeliness by dropping multiple data types together at the same increments for each data quality domain. However, there could be alternative versions of compounded simulation where diagnosis, medication, and laboratory data qualities are induced at different levels simultaneously; this may be of interest for future research. Seventh, we assessed the robustness of the phenotypes as they were designated. However, future research could stratify the analysis by the severity of the condition of interest, as severity could impact diagnosis, medication, and laboratory coding behavior and quality. And lastly, our study did not consider any existing data quality thresholds to assess epidemiological impacts on T2D patients using EHRs (eg, comparing national rates of T2D in a neighborhood vs T2D rates identified using EHR data of patients residing in the same neighborhood).
As a result of these observations, we believe that there is a growing need in the United States for a standardized phenotype definition to identify T2D populations while considering the challenges of data quality issues in real-world data, such as from EHRs. The universal phenotype definition should have the ability to integrate features of EHR data while being resistant to common data quality issues. Such a phenotype definition ought to also consider factors that may introduce racial bias and disparities, which eventually may result in health inequities. And lastly, this fundamental definition must also consider integration and interoperability with other data sources, such as claims data, and alignment with existing data interoperability standards. Our research hopes to inspire T2D subject matter experts, at least, to begin conversations toward creating a universal definition for a disease that is extremely common. That said, there is opportunity for additional research that ties issues of data quality with those of phenotypes, data types, and data sources.
Our research provides novel results to understand the effect of data quality issues in data sources like EHRs to identify T2D population-level groups of interest using commonly available T2D phenotype definitions. The study results can inform research or operational efforts using large clinical data repositories to identify T2D populations. Lastly, the study findings can inform efforts to consolidate T2D phenotypes in the near future.