We identified and reviewed all known apps in public-facing marketplaces of the top 10 EHR vendors in the United States, which include (in order of market share) Epic Systems Corporation; Allscripts; eClinicalWorks, LLC; NextGen Healthcare; GE Healthcare; athenahealth, Inc; Cerner Corporation; Greenway Health LLC; Practice Fusion; and eMDs [17]. GE Healthcare did not have a public facing app marketplace at the time of this writing and Practice Fusion was recently acquired by Allscripts [18]. The leading vendors in the US market were chosen for analysis because they had publicly available app marketplaces (with the exception of GE Healthcare), and they represented a cluster of vendors serving the majority of a common set of customers.

We used custom web scrapers and public ReST (Representational State Transfer) endpoints from these EHR marketplaces to gather EHR app data such as name, description, links, website, EHR-defined app categories, ratings, reviews, EHR versions, and other available information. This information was recorded in our EHR app database wherein we consolidated duplicate EHR apps that were listed in multiple EHR marketplaces. The last data extraction occurred in February 2019. Clear indication of FHIR compatibility was not consistently present in the extracted data within or across EHR marketplaces. Marketplace offerings that offered professional services without clear evidence of EHR integration (eg, website builders or marketing services), teams of professionals granted access to EHR interfaces (eg, offsite medical coders), and medical devices without EHR integration (eg, stand-alone weight scale) were not considered apps and were excluded.

Two informaticists (authors JR and BW) used grounded theory principles to create categories that emerged from the EHR app information [19]. EHR app classifications were created inductively by each reviewer independently based upon available information about each EHR app. Importantly, not all data fields were available across all EHRs. For example, not all marketplaces included information indicating whether the app was open source or a commercial offering. Even though some marketplaces included this information, apps were generally classified based on the information available that was common across apps from all marketplaces. When the EHR app information was either inadequate or missing, making it difficult to accurately classify the app, we referenced the app developer’s website. As common EHR app features, functions, or purposes emerged, we created categories to group similar apps. Where similarity between categories existed, they were related to form larger, more inclusive categories. Conversely, if sufficient EHR app divergence existed within a category, we subclassified the apps into more unified categories [20]. We drew category names from app descriptions, EHR-designated classifications, and common industry concepts. A minimum of three apps were required to form a category.

Following the review and initial classification by each reviewer independently, a joint review process commenced between the two reviewers. The reviewers worked together to come to a consensus on category names, the organization of the taxonomic class hierarchy, and the correct classification of each EHR app. To facilitate the joint review process, a database with all EHR app information as well as notes from each reviewer and their initial classifications were utilized. Each EHR app was reviewed and discussed together. As consensus categories emerged, formal definitions for each category were created and refined. A final classification was used to denote the consensus classification. Consensus was reached when both reviewers agreed with the classification. In cases where the reviewers failed to reach consensus, a third-party arbitrator was available. Through multiple rounds of discussion and debate, a consensus EHR app taxonomy emerged.

Of the eight EHRs with public-facing marketplaces, we identified a total of 749 offerings. The total number of offerings for marketplaces ranged from 21 (eMD) to 227 (Athenahealth). In total, 153 offerings were listed on at least two EHR marketplaces, resulting in a total of 596 unique offerings; 125 were excluded from consideration for not meeting our inclusion criteria, which resulted in 471 unique apps being incorporated into our taxonomy. We categorized the EHR app into a taxonomy consisting of 3 primary categories, 15 secondary categories, and 55 tertiary categories. The three primary categories were administrative (n=203 apps, 43.1%), provider support (n=159, 33.8%), and patient care (n=109, 23.1%).

For each EHR, the distribution of apps across the primary categories followed a similar trend. In general, administrative apps make up the greatest portion of EHR apps, followed by provider support apps and then primary care apps. Interestingly, Cerner’s marketplace has a higher ratio of patient care apps, followed by provider support and then administrative. There was also a large variability in the number of listings excluded for not meeting criteria for being an app between EHR marketplaces, with eClinicalWorks accounting for 65 of 125 (52%) excluded offerings (Table 1). Each primary category is described in further detail below.

We conducted a systematic search and analysis of apps in EHR app marketplaces to help organize, categorize, and characterize available EHR apps. This study brings value to the health IT industry because it creates a common vocabulary that can be used to communicate about EHR apps, helps health care organizations identify EHR app solutions, and justifies investment into the research and development of new EHR apps. With this study, we can identify common patterns of EHR integration approaches and create a template to streamline future EHR app development and integration. This helps researchers identify gaps in integration capabilities, standards, or functionalities that need to be addressed by governing bodies, standards organizations, EHR vendors, or EHR app developers.

Our EHR app review organized and characterized 471 unique EHR apps into 3 primary categories, 15 secondary categories, and 55 tertiary categories. Several categories were larger or more well-defined than others. Administrative apps represented the largest share of EHR apps with 203 apps. Provider support and patient care apps were the other primary categories with 159 and 109 EHR apps, respectively. EHR marketplaces tended to reflect this overall trend with the majority of apps falling under the administrative category, followed by provider support and patient care. Cerner followed a distinctly different trend with patient care representing the majority of their apps and only a single administrative app. This may in part be attributed to how Cerner validates apps that are submitted by third-party developers. While all app galleries reviewed here have a submission process, and several list disclaimers that not all submissions may be listed upon submission, Cerner has an additional validation step. Apps that don’t meet a certain standard set by Cerner may be rejected or asked to resubmit after outstanding issues are resolved. This adds extra rigor in the Cerner app submission process that may account for the lower total app count in their gallery as well as the different ratio of primary categories observed. The fact that zero offerings in Cerner’s gallery were considered “Not an app” and excluded from consideration in our search and analysis may be attributed to their unique validation approach and, generally speaking, Cerner’s apps required less attention when assigning apps to categories. However, Cerner also had fewer total offerings listed in their gallery (n=28) than all other vendors except for eMD (n=21). App quality is an important issue and while the right amount of validation is difficult to quantify [8], it is important to note that if validation is too strict, it could suppress innovation and defeat a key purpose of the app model, which allows competition among developers based on app value and price [9]. This allows clients to validate app offerings and reward innovation and the value the apps provide [15].

Interestingly, provider support apps had the greatest variability and ambiguity among the three primary categories of apps. Provider support had more secondary categories than the other two primary categories combined, accounting for 8 of the 15 (53%) total secondary categories. Many apps offered multiple functionalities or had feature sets that made it difficult to assign secondary and tertiary categories. The value these apps provided and how they were intended to be used by the provider was often unclear. Provider support accounted for 63% (14/22) of apps that were not specific enough to place in a secondary category. Additionally, provider support accounted for 46% (25/54) of apps that were not specific enough to place in a tertiary category. The secondary categories under provider support with the most apps without tertiary categories were specialty EHR (6/12, 50%), EHR efficiency (7/16, 44%), and medical device integration (7/13, 39%). Among all other secondary categories, office device integration had the highest percentage of apps that did not fall into a tertiary category (4/17, 24%), followed by clinical decision support (7/40, 18%). This suggests that the provider support category has the greatest need for further refinement and innovation out of the three primary categories.

The app model is a remedy to the one-size-fits-all strategy that is failing to meet the needs of patients, providers, and administrators in a rapidly evolving landscape [8]. For the app model to be effective, the apps listed need to solve a clearly defined problem instead of offering diverse sets of features and functionalities that begin to approximate a one-size-fits-all-solution. In other words, it needs to be obvious what category the app belongs in. The taxonomy of apps we have curated will help health IT companies and app developers match app development to a well-defined purpose and assist health professionals in identifying gaps in the current set of app categories. As app functionality and feature sets become more cohesive and achieve alignment with specific EHR app categories, the value and impact the app model will have on health care will increase as patients, providers, and administrators can more easily search, install, and ultimately be the market force that will drive innovation and value of EHR apps.

We acknowledge several shortcomings of the current review. First, our review is limited only to apps currently available on the top 10 EHR app marketplaces by market share in the United States. We acknowledge there are other EHR vendors worldwide developing app marketplaces as well that were not reviewed here. During our review and research of EHR apps, we came across several other apps that claimed EHR integration that would have been included if they had been listed in an EHR marketplace. It would be unfeasible to know all apps that integrate with EHRs; nevertheless, we hope that as EHR marketplaces mature, these apps will become listed in the EHR marketplaces and organized in our review. Second, several apps had characteristics or features that could justify their classification under more than one category. In these cases, we endeavored to classify apps to the lowest level in the taxonomy as possible while still accurately reflecting the apps’ primary purpose, which sometimes resulted in higher level classifications. In a few instances, when the information was insufficient to determine whether an offering was an app, we erred on the side of inclusion. As a result, a few apps in our taxonomy may have been inappropriately included. As further information becomes available, their inclusion and classification will be re-evaluated. Third, the review was conducted by two informaticists. We recognize that shortcomings, inaccuracies, and/or bias may exist in the interpretation and characterization of the apps. Independent input from a panel of expert stakeholders would increase robustness and validity of the EHR app review. Finally, the EHR app review represents a single point in time (February 2019). However, as new EHR apps get added to marketplaces and new app information becomes available, the results will become outdated. We anticipate conducting this review again in a few years to understand how EHR app marketplaces have evolved over time.

The SMART app model was proposed in 2009 by Mandl et al [9]. Since then, EHR vendors have followed suit by building their own app marketplaces. We identified hundreds of apps in these marketplaces that allow integration with their respective EHR vendor. Current EHR marketplaces do not fully reflect the original proposal made by Mandl et al [9], which called for total substitutability of apps across any EHR by conforming to a single standard. Without conforming to a single standard as proposed by Mandl et al [9], each app would need to integrate with each EHR marketplace individually, as is the case today. For instance, we observed that 153 apps integrate with two or more EHRs. This does not quite meet the proposal made by Mandl et al [9] where an app lives on a single platform and can be integrated with any health system regardless of EHR vendor.

We characterized and organized a diverse and rich set of third-party EHR apps. This work provides an important reference for developers, researchers, and EHR customers to more easily search, review, and compare apps in EHR app marketplaces. While future research and validation among independent informaticists and stakeholders will increase the validity and value of this review, this work provides a strong foundation upon which future EHR app research will be established.