The RAND-modified Delphi method was used to develop QIs for the use of EMRs from January 2024 to December 2024 based on the method used by Van den Bulck et al [27-29]. The process included the following: (1) development of a questionnaire based on recommendations and existing QIs from evidence-based guidelines and (2) rating procedure by an expert panel in 3 rounds, namely individual written questionnaire, online consensus meeting, and final appraisal.

The TRIP database and MEDLINE were searched for guidelines and indicator sets using the following search terms: “Electronic medical records” AND (“Primary Health Care” OR “Primary Care” OR “General Practice”) AND (“Standard of Care” OR “guideline” OR “recommendation” OR “quality indicator” OR “quality of health care” OR “quality of care”). The search was also extended to Google Scholar (accessed on April 2, 2024). The sources were selected by the primary researcher using the criteria specified in Textbox 1. Sources were not filtered based on the quality of the methodology because of the limited methodological information in any of the sources found in the first search. An exhaustive list of all indicators and recommendations relevant to EMR quality was compiled. For inclusion, indicators and recommendations were filtered using the specific, measurable, acceptable, realistic, and time-bound (SMART) principle [27-29]. A QI is defined by Grypdonck et al as “a measurable tool to assess the quality of care, including outcome-, process- and patient-oriented indicators” [30]. For recommendations, the following definition provided by the World Health Organization was used: “Recommendations are statements designed to help end-users make informed decisions on whether, when and how to undertake specific actions such as clinical interventions, diagnostic tests or public health measures, to achieve the best possible individual or collective health outcomes” [31]. Screening was performed by 2 independent researchers: a GP (primary researcher) and a methodologist. If there was a difference in the results for an indicator or recommendation between the 2 researchers, they discussed it until a consensus was reached. All remaining items (indicators and recommendations) were then categorized into topics according to the organization of the PHS in Belgium as explained by Domus Medica [19], and comparable with categories defined by Jabaaij et al [13] in the EMR scan for general practitioners: completeness and adequacy of the problem list, encoded registration in the EMR, completeness and actuality of medication list, risk factors and drug monitoring, patient identification and contact information, vaccination status, and patients’ choices.

GPs and EMR developers were invited to participate in the panel via email. Using a purposive sampling technique, priority was given to representing all involved parties at the regional (Flanders, Wallonia, and Brussels) and occupational levels (solo GPs, GPs in group practices, programmers, and academic personnel). The final panel consisted of 18 participants: 11 (61%) Flemish-speaking participants and 7 (39%) French-speaking participants. There were 12 GPs (recently retired: n=1, 6%; worked in a community health center: n=1, 6%; had an individual practice: n=2, 11%; and worked in a group practice: n=8, 44%) and 6 (33%) EMR developers from different EMRs. They were provided with information about the aims and methodology of this study.

Each panel member received an individual feedback report 2 weeks before the consensus meeting. It also included new recommendations and comments. The report contained all items with a color code representing their potential to reach a consensus (Multimedia Appendix 1).

An online consensus meeting was held using Microsoft Teams [34]. To improve the efficiency of the consensus meeting, focus was given to the items categorized as “uncertain,” together with the newly added indicators proposed by the panelists. Items with high potential were automatically selected for the final selection unless panel members requested a decision-making discussion. Items with low potential were excluded from the final selection unless the panel members requested deliberation.

All accepted indicators and recommendations were adjusted to the conclusions of the consensus meeting and conformed to the SMART criteria [27-29]. The recommendations were transformed into QIs as percentages. The final report was sent to the panel members for final acceptance in Dutch and French. The final indicator set was translated into English by the researchers.

This study was approved on June 25, 2024, by the Social and Societal Ethics Committee of KU Leuven (G-2024-8020-R2(AMD)). The project was also vetted and approved by KU Leuven’s Privacy and Ethics platform in view of the principles and obligations laid down in the General Data Protection Regulation of the European Parliament and the Council of 27 April 2016 [35]. Participants were asked to digitally sign an informed consent form before participation. All data were pseudonymized after the written questionnaire round using a unique identifier. Personal information was separated from other research data in a second file with password protection. The personal data was used to provide the participants the feedback report, to invite them for the consensus meeting round, and to approve the final report. All personal information was deleted after the study was finished. No compensation was provided to the participants.

A total of 4 review articles listed QIs for the use of EMRs, containing 27 QIs [13,15,36-38]. Nine guidelines contained 53 QIs and 125 recommendations [11,19,20,39-42]. Where possible, duplicate items were removed, and indicators were linked to similar recommendations in the guidelines. This resulted in a combined list of 20 QIs and 30 recommendations (Figure 1 [11,13,15,19,20,36-42]; Multimedia Appendices 2-4).

All panel members (18/18, 100%) completed the questionnaire. On the basis of these categories, out of 50 items, 21 (42%) were selected, 27 (54%) required further discussion, and 2 (4%) were excluded.

Of the 18 panel members, 13 (72%) were present for the consensus meeting—9 (50%) GPs and 4 (22%) EMR developers—8 (44%) of whom were Flemish speaking and 5 (28%) French speaking. In the panel round, none of the preselected indicators (n=7) or recommendations (n=14) were discarded. Of the 10 indicators labeled as “discussion,” 7 (70%) were selected, and 1 (10%) indicator was further divided into 7 specific indicators (Multimedia Appendix 1; indicator 3). Of the 17 recommendations labeled as “discussion,” 6 (35%) were selected. This resulted in a list of 20 indicators and 20 recommendations (Multimedia Appendix 1). During the consensus meeting, some items that were not yet extractable were included because the panel members felt that their clinical importance should prompt EMR providers to adapt their software.

The 20 recommendations were transformed into QIs and added to the 20 other QIs, resulting in 16 (40%) QIs regarding the completeness and adequacy of the problem list, 4 (10%) QIs regarding encoded registration in the EMR, 3 (8%) QIs regarding the completeness and actuality of the medication list, 6 (15%) QIs regarding risk factors and drug monitoring, 4 (10%) QIs regarding patient identification and contact information, 3 (8%) QIs regarding vaccination status, and 4 (10%) QIs regarding patient choices (Textbox 2; Multimedia Appendix 5). All panel members present in the consensus meeting (13/13, 100%) approved the final report.

This study used the RAND-modified Delphi method to validate a set of 40 QIs for the correct use of EMRs by GPs based on the indicators and recommendations found in the literature. These indicators are (potentially) EMR extractable and evaluate key aspects of EMR use by GPs: completeness and adequacy of the problem list, encoded registration, completeness and actuality of the medication list, risk factors and drug monitoring, patient identification and contact information, vaccination status, and patient choices. Using the SMART principle, we ensured that the QIs could be evaluated in clinical practice [27-29,44-46].

Previously, Jabaaij et al [38] developed the EMR scan for general practitioners in the Netherlands to evaluate EMR use. However, the method they used to define QIs was unclear. Ngugi et al [18] developed and validated a set of indicators for the use of EMRs in low- and middle-income countries. Their QIs mostly evaluated the general use of the EMRs (eg, indicator 7, “% of required data elements contained in EHRs”). Our study focuses on how data are registered and encoded in EMRs, focusing on which data are registered [18]. Other indicator sets were either outdated (aged >20 years) [36], focused on clinical performance instead of use of the EMR [15], or evaluated EMR use on the mesolevel or macrolevel instead of the microlevel [37]. Hamade et al [17] noted that while there are many studies on the implementation of EMR, there is a scarcity of studies on its use.

Most of the selected indicators evaluated the completeness and adequacy of the problem list. The problem list provides important information for decision-making [47]. Our indicators are primarily process indicators because they focus on the process of EMR registration (ie, use). However, these QIs cannot be directly linked to patient outcomes. Process indicators are useful for quality assessment. They are more sensitive to differences in quality than outcome measures and are easier to interpret [48]. Recent literature shows an interest in eA&F [22,29], dashboard interface features to support reflection on practice (“barometers”) [27-29], and computer-based decision support in primary care [49]. The quality of data in EMRs impacts the effectiveness of these systems. Several studies have shown that improving data quality may improve the quality and efficiency of care and reduce mortality [4-6]. Unfortunately, data show that physicians spend approximately two-thirds of their work time interacting with EMRs in their offices. This administrative burden negatively impacts clinicians’ wellness, leading to an increased chance of burnout. Recently, the interest in the use of artificial intelligence for analyzing data, identifying hidden information, identifying risks, and providing suggestions for diagnosis has increased rapidly. This can greatly facilitate the process for GPs to easily improve data quality without spending more time entering data in the EMR and less time interacting with patients [50]. GPs should not need to adapt their working habits to the capabilities of their medical software; rather, EMRs should be adapted to the needs of GPs to facilitate the correct registration of data [51]. An important lever to encourage EMR developers to align their software more closely with clinicians’ data entry and retrieval needs lies in regulatory frameworks, such as homologation. This is a certification procedure to evaluate the mandatory standards and certification requirements that EMRs must fulfill to be approved for use within the health care system [52]. By embedding QIs derived through expert consensus, such as those developed in this study, into these frameworks, regulators can create explicit performance benchmarks that EMRs must meet. This alignment incentivizes developers to design functionalities that facilitate accurate, complete, and efficient data capture and reporting by clinicians [50].

Our panel included members working in all parts of Belgium and in all types of practices, giving it broad support. Because our panel included health professionals and EMR developers, we were able to define clinically relevant indicators that considered the technical requirements for extractability. During the consensus meeting, the EMR developers emphasized that these indicators could possibly stimulate GPs to use the features they have already developed more extensively. We are convinced that their opinion was a valuable addition to the panel. EMR developers were the most capable of assessing whether a QI could be automatically extracted from the EHR data. However, we cannot exclude a potential bias caused by their inclusion (eg, the ease or cost of addressing a recommended QI outweighing the importance of making that QI available from a public health perspective). The scarcity of literature on the initial set of QIs and recommendations is a double-edged sword for our study. On the one hand, it limits the strengths of the initial set of QIs and recommendations because none of our sources used a systematic approach to evaluate the validity of their QIs and recommendations, had a well-explained methodology [13,15,36-38], or could be evaluated using the Appraisal of Guidelines for Research and Evaluation II method [11,19,20,39-42]. On the other hand, this allowed us to conduct a broad and general search on the topic, which could increase its universal applicability. There are many differences between countries in terms of how and which data are registered [11,19,20,39-42]. We attempted to generalize the QIs and recommendations as much as possible. However, adaptation to the local context is necessary. Another important limitation is that some indicators are not currently extractable. However, as mentioned earlier, these indicators reflect aspects of clinical practice that are important enough to warrant adaptation of EMRs to include this information.

This set of indicators is important for future research on EMR use. Developing a list of QIs is the first step in implementing a strategy and realizing a quality loop. The next essential step is to conduct practice testing for operational validity [29]. These results can be used to create an eA&F system [22,23]. This set of QIs can also be used to develop specific (computer based) training for GPs based on a set of learning goals [2]. It should be noted that our QIs can be used for the correct use of EMRs but cannot be used to quantify the quality of care provided by GPs. Further research is necessary to evaluate whether the correct use of the EMR, as proposed in this set of QIs, also affects the quality of care provided to patients and its possible effect on the data in the PHS.

This study provided a set of 40 EMR-extractable QIs for the correct use of EMRs in primary care based on international guidelines and approved by GPs and EMR developers. These QIs can be used as a framework to measure and improve the quality and completeness of EMRs in primary care.