RISE is a national registry that passively collects data from EHRs of participating practices, aggregates and centrally analyzes performance on quality measures in rheumatology [12]. RISE practices have an on-site “registry connector” that uploads EHR data to the RISE clinical data warehouse on a nightly basis. Data that are uploaded include information regarding patient diagnoses, medications, laboratory studies, and vital signs.

When a practice first joins RISE, practice personnel work with registry staff to map structured data elements to relevant quality measures. Occasionally, practices request data elements to be pulled from clinical notes. This is feasible when data are recorded in a highly reliable, “semi-structured” format. Data elements extracted from the EHR are used to calculate electronic quality measures for submission to Centers for Medicare & Medicaid Services (CMS) as part of national pay-for-performance programs. For example, patients with RA are identified using ICD codes to enter a denominator population. DA scores are extracted (usually from structured fields) to determine whether denominator patients meet the criteria established by quality measures. Practice, provider, and patient-level performance on quality measures is fed back to providers through a web-based dashboard. Patient-level EHR data that include all variables mentioned above, in addition to quality measure denominator and numerator information is provided to RISE Data Analytic Centers for analysis. Additional details on the structure and function of the RISE registry are described elsewhere [12].

As of December 2018, RISE held validated data from 1113 providers in 226 practices, representing approximately 32% of the US clinical rheumatology workforce. RISE can connect to most certified EHR systems in the United States; as of 2018, the registry could map to over 30 different EHRs used by rheumatologists. EHRs largely reflect those used by community rheumatologists, as academic medical centers (and therefore large EHR vendors such as Epic and Cerner) are underrepresented in the registry.

We analyzed data on individuals with RA seen in rheumatology practices participating in RISE between January 1, 2018, and December 31, 2018. Patients included in this study were 18 years of age or older and had ≥1 International Classification of Diseases Clinical Modification, Ninth or Tenth Revision (ICD-9-CM or ICD-10-CM) code for RA with at least 1 clinical face-to-face encounter in 2018. These inclusion criteria are based on the denominator definitions for the quality measures used to calculate the outcomes of interest (see section below).

We selected two measures—routine assessment of DA and FS—for patients with RA since these are key components of a treat-to-target approach and among the newer process measures introduced in the national pay scheme for performance programs, specifically for rheumatologists. The American College of Rheumatology is currently collaborating with the National Quality Forum (NQF) to develop outcome measures on the basis of DA and FS. Thus, it is especially important to understand variations in the collection of these measures and any potential factors influencing their documentation. Assessment and documentation of DA and FS outcomes were assessed at the practice-level by calculating the performance on NQF-endorsed quality measures. Performance on each measure was defined as follows: (1) DA: percentage of patients aged 18 years and older with a diagnosis of RA, whose DA was assessed using a standardized measurement tool at 50% or more face-to-face encounters for RA during the measurement period [13]; and (2) FS: percentage of patients aged 18 years and older with a diagnosis of RA, whose FS was assessed using a standardized measurement tool at least once during the measurement period [14]. The measurement period was the 12-month period between January 1 and December 31, 2018.

Covariates included EHR system (NextGen, eClinicalWorks, GE Centricity, eMDs, Allscripts, Amazing Charts, Aprima, others included Lytec MD, Medent, Medisoft, Raintree System IC, MD office, Integrity, Carecloud, MedTrio, Greenway/Primesuite, iPatientCare, Prime Clinical System, MacPractice MD, IMS, SRS EHR, PrognoCIS, Cerner, Practice Fusion, DrChrono, Chart Maker Clinical, STI, American Medical Software, Athena Clinicals, Praxis EMR, RheumDocs, Greenway Intergy, Athena UniCharts, and ChartLogic) and a variety of practice and patient characteristics previously associated with measure performance. Practice characteristics included the number of providers within the practice; practice type (single-specialty group practice, solo practitioner, multi-specialty group practice, other clinical settings, and large health system); and geographical region in the United States (Northeast, Midwest, South, and West). Practice-level sociodemographic variables were calculated by aggregating the characteristics of patients included in the study and included the proportion of patients aged ≥65 years, females, non-White individuals, and of those with noncommercial insurance. Patient characteristics included age, sex, race/ethnicity, insurance type (private, Medicaid, Medicare, or other), and Charlson comorbidity index (CCI) score calculated in accordance with the Deyo modification based on codes reported at any time during the study period [15].

To learn more about potential reasons for differences in performance on DA and FS measures across EHR systems, we also assessed documentation workflows among a subset of RISE practices. A survey was disseminated electronically using a commercial survey web application to the RISE practices’ providers and administrators between November 11, 2020, and April 14, 2021. The survey included 9 questions (Multimedia Appendix 1), covering the topics of practice characteristics (3 questions), and EHR system–related factors, such as the presence of a rheumatology-specific module or dedicated structured fields for PROs, which might influence DA and FS documentation workflows (6 questions).

We used descriptive statistics to summarize patient and practice characteristics. Multivariable linear regression was used to examine the independent effect of EHR system on practice-level performance. We used zero-inflated models when the occurrence of zeros for practice-performance was meaningful (27.7% for DA; 40.4% for FS). These models allow for modeling of overly dispersed data. The outcome variable for the zero-inflated analyses was the count (rate) of patients in a practice, who received recommended care [16]. Zero-inflated Poisson (ZIP) and zero-inflated negative binomial (ZINB) models, all adjusted for potential confounders, were compared using Akaike’s Information Criterion (AIC), Bayesian Information Criterion (BIC) values, and log-likelihood to assess the goodness of fit. The incidence rate ratio (IRR) for each of the predictor variables in both count (rate) and logit parts of the model was reported along with 95% CIs and P values.

All the models (linear and zero-inflated) were adjusted for practice characteristics (including practice type, size, and geographical region) and patient case-mix (including patient age, sex, race, and insurance) since these variables have been previously shown to have a significant association with performance on rheumatology quality measures [5,17], and our goal was to isolate the impact of the EHR vendor on performance. To account for differences in case-mix, we adjusted for the aggregate characteristics of patients seen in the practice (proportion of patients aged ≥65 years, proportion of females, proportion of non-White patients, and the proportion of patients with noncommercial insurance). Missing values were included in the analyses as their own separate category without imputation. For all analyses, P values less than .05 were considered statistically significant.

All analyses were performed using STATA statistical software (version 16, StataCorp). This study was approved by the Western Institutional Review Board, Inc. as well as the Committee on Human Research at the University of California, San Francisco.

Our practice-level analysis included 314,793 individuals with RA who met the inclusion criteria across 220 practices in this study; 6 practices had no patients who met the inclusion criteria and were therefore excluded. Among all included patients (N=314,793), the mean age was 62.0 (SD 14.3) years, 76.1% were female, 67.7% were White, and 31.8% had private insurance. Most practices were single-specialty group practices (56.8%). NextGen was the most commonly used EHR system (34.1% of practices), followed by eClinicalWorks (14.5%) and Amazing Charts (9.5%) (Table 1).

Overall, median (IQR) practice-level performance on DA and FS quality measures was 30.1 (0, 74.8) and 9.0 (0, 74.2), respectively. Figures 1 and 2 demonstrate the differences in quality measure performance on the DA and FS measures for practices using different EHR systems.

In unadjusted analyses, practices using NextGen showed significantly higher performance on DA and FS measures compared to practices using other EHR systems (Table 2). In multivariate linear regression analyses adjusting for practice characteristics and patient case-mix, practices that used NextGen had higher performance. Specifically, NextGen practices performed significantly better than Allscripts on the DA measure (51.4% vs 5.0%) and significantly better than eClinicalWorks and eMDs on the FS measure (49.3% vs 29.0% and 10.9%, respectively; Table 2). Full models with parameter estimates for practice and case-mix variables are included in Multimedia Appendix 2.

Marginal means were estimated using multivariate regression models. Confidence intervals of <0 were truncated at 0.

Because we found a significant number of practices with zero performance (27.7% for DA; 40.4% for FS), we also used zero-inflated models to analyze the association between EHR systems and DA and FS documentation (Multimedia Appendix 3). Zero-inflated negative binomial models revealed that the differences in performance across EHRs were driven largely by the practices with absent documentation of DA and FS (zero performance): although we found no differences in the count portion of the ZINB model across EHRs, there were significant differences between NextGen versus other EHRs in the logit portion of the model. For example, practices using Allscripts had approximately a 2.5 times higher rate of having zero performance on DA and FS compared to practices that used NextGen (Table 3). On the other hand, practices that used GE Centricity were less likely to have zero performance on DA than practices that used NextGen (P<.01).

Finally, among 40 (18.2%) practices with survey responses, NextGen was the most commonly used EHR system (37.5%), followed by eClinicalWorks (22.5%) and Amazing Charts (7.5%); other EHR systems accounted for 32.5%. The majority of the responding practices using NextGen (93.3%) reported that they relied on structured data fields for DA and FS quality measure documentation in the EHR. Conversely, the vast majority of the non-NextGen practices reported that clinicians documented DA and FS in clinical notes. After the survey was closed, we additionally queried survey respondents to understand local workflows. For example, we found that NextGen includes rheumatology-specific templates that facilitate documentation of RA outcomes and functionality to track this information over time. In contrast, those using Amazing Charts enter DA and FS measures in a semistructured way (ie, in the same section of the note for every patient). This workflow, although it departs from the structured fields used by most NextGen practices, allows the registry vendor to manually extract these data for national performance reporting but is not amenable to tracking outcomes over time.

Although quality measures are often used to evaluate the performance of individual clinicians or health systems, the impact of health information technology on performance remains extremely understudied. We used a unique data source, the ACR’s RISE registry, which captures data from rheumatology practices across the United States, to investigate the relationship between performance on nationally endorsed RA quality measures and the EHR system used by practices.

We found that after adjusting for both practice characteristics and patient case-mix, performance in practices using some EHR systems was consistently higher; the EHRs with the highest performance generally had rheumatology-specific templates or modules in their foundation software, which facilitated collection and tracking of key RA outcomes. These findings raise important questions about the role of EHR vendors in creating software that facilitates high quality of care in rheumatology.

In both rheumatology and more general practice, studies that formally assess the impact of health information technology systems, including EHRs, on quality measure performance are limited. In a prior study using the RISE registry, we found that NextGen practices were able to improve performance on quality measures more rapidly over time than practices with other EHR systems [17]. Literature is emerging to support the notion that EHR systems can be an important factor in quality of care; for example, one study investigated a large group of primary care practices with different EHRs and their frequency of unsafe prescribing of cyclosporine, tacrolimus, and diltiazem and found important differences across EHRs [18]. In the case of RISE practices, NextGen practices reported the availability of rheumatology-specific templates in the EHR foundation software to capture key RA outcomes, facilitating quality measurement and disease-tracking. Practices using EHRs that lack this feature are much less likely to document effectively, although some of them have managed to find other methods (eg, semistructured data collection in clinical notes) for increasing accurate registry data measures for use in quality measure calculations. However, this type of documentation is less conducive to chronic disease management since it renders longitudinal tracking of disease outcomes using the EHR challenging.

Using current technology, documentation of key RA outcomes in structured EHR data fields remains the most feasible way to ensure accurate data capture and quality measurement. However, in the future, it is possible that extraction of information from clinical notes will become easier. The application of advanced approaches using text searches, natural language processing algorithms, or machine learning to extract information about disease outcomes directly from clinical notes could become more feasible in the future [19], although these strategies have largely not been demonstrated to be reliable in nonresearch settings to date and are often very cost prohibitive [20]. On the other hand, our study illustrates that is feasible in the short-term for EHR systems to modify their foundation software to include content that facilitates rheumatology practice, including capture of key RA outcomes such as DA and FS.

Further, although the idea is provocative, we think it is time for health care quality measurement to consider the range of factors that influence performance. Using a broader quality measurement paradigm, EHR vendors, in addition to individual physicians or health systems, should share in incentives or penalties associated with quality measure performance for chronic diseases such as RA [9,21,22]. Although in theory, a marketplace with multiple competitive EHR systems might have resulted in lower prices and increased functionality across all systems, in reality, the costs of switching EHR systems is high, and the risk of downtime or loss of historical data makes changing EHR vendors difficult. A shared incentive model could encourage EHR vendors to implement tools to support quality measurement and improvement. Others have explored this idea of shared responsibility for performance, particularly in the realm of patient safety [23]. For example, poorly designed user interfaces can reduce clinicians’ access to key data needed to ensure patient safety [24]. In a model designed around shared responsibility, EHR vendors would be incentivized to incorporate a user-centered process in designing and building software to support clinicians in meeting key quality metrics. Whether around patient safety or management of chronic diseases like RA, this would require EHR vendors to develop ongoing strategies to track the usability of their product and to set up systems for innovation and continuous quality improvement of software [25]. Public and private agencies could be charged with creating programs to incentivize ongoing usability-testing and to track resulting measure performance [26]. Such a model would, of course, require the full collaboration of practices and clinicians, who would need to incorporate such usability testing into their workflows and be willing to alter existing workflows to take advantage of software improvements.

Our study has important strengths. Data were collected passively from different EHR systems and includes all patients with RA who met criteria for the denominator of the quality measure at each practice, thus greatly reducing the risk of selection bias inherent in other study designs. Moreover, the large number of practices and EHR systems represented in the RISE registry provides a unique view of a range of rheumatology practices using different EHR systems across the United States. Along with these strengths, this study has some important limitations. Information obtained from the EHR reflects care documented rather than care delivered. Clinicians may have assessed DA or FS for their patients but failed to document them or documented them in ways that are not easily retrievable by the registry; however, since our intention was to examine capture of data in EHRs, which are used to quality measure performance, this limitation does not impact our conclusions. A second, important limitation, is that we were limited to studying the EHRs of practices that participate in RISE. The registry currently has limited coverage of some EHRs with a greater market share among academic practices in the United States (eg, Epic and Cerner) and covers only an estimated 32% of the US clinical rheumatology workforce. Further research is needed to assess the relationships between EHR systems and quality measure performance, especially in academic settings.

In summary, this study shows a strong relationship between the EHR system used by practices and performance on DA and FS quality measures for RA. Future research should investigate whether features of EHRs, which facilitate documentation and tracking of RA outcomes, facilitate improved outcomes among patients with RA over time. Developing rheumatology-specific standards across EHRs may promote routine collection of RA measures, which, in turn, could improve RA outcomes.