Applying STOPP Guidelines in Primary Care Through Electronic Medical Record Decision Support: Randomized Control Trial Highlighting the Importance of Data Quality

Background Potentially Inappropriate Prescriptions (PIPs) are a common cause of morbidity, particularly in the elderly. Objective We sought to understand how the Screening Tool of Older People’s Prescriptions (STOPP) prescribing criteria, implemented in a routinely used primary care Electronic Medical Record (EMR), could impact PIP rates in community (non-academic) primary care practices. Methods We conducted a mixed-method, pragmatic, cluster, randomized control trial in research naïve primary care practices. Phase 1: In the randomized controlled trial, 40 fully automated STOPP rules were implemented as EMR alerts during a 16-week intervention period. The control group did not receive the 40 STOPP rules (but received other alerts). Participants were recruited through the OSCAR EMR user group mailing list and in person at user group meetings. Results were assessed by querying EMR data PIPs. EMR data quality probes were included. Phase 2: physicians were invited to participate in 1-hour semi-structured interviews to discuss the results. Results In the EMR, 40 STOPP rules were successfully implemented. Phase 1: A total of 28 physicians from 8 practices were recruited (16 in intervention and 12 in control groups). The calculated PIP rate was 2.6% (138/5308) (control) and 4.11% (768/18,668) (intervention) at baseline. No change in PIPs was observed through the intervention (P=.80). Data quality probes generally showed low use of problem list and medication list. Phase 2: A total of 5 physicians participated. All the participants felt that they were aware of the alerts but commented on workflow and presentation challenges. Conclusions The calculated PIP rate was markedly less than the expected rate found in literature (2.6% and 4.0% vs 20% in literature). Data quality probes highlighted issues related to completeness of data in areas of the EMR used for PIP reporting and by the decision support such as problem and medication lists. Users also highlighted areas for better integration of STOPP guidelines with prescribing workflows. Many of the STOPP criteria can be implemented in EMRs using simple logic. However, data quality in EMRs continues to be a challenge and was a limiting step in the effectiveness of the decision support in this study. This is important as decision makers continue to fund implementation and adoption of EMRs with the expectation of the use of advanced tools (such as decision support) without ongoing review of data quality and improvement. Trial Registration Clinicaltrials.gov NCT02130895; https://clinicaltrials.gov/ct2/show/NCT02130895 (Archived by WebCite at http://www.webcitation.org/6qyFigSYT)


1a-i) Identify the mode of delivery in the title
Identify the mode of delivery. Preferably use "web-based" and/or "mobile" and/or "electronic game" in the title. Avoid ambiguous terms like "online", "virtual", "interactive". Use "Internet-based" only if Intervention includes non-web-based Internet components (e.g. email), use "computer-based" or "electronic" only if ofoine products are used. Use "virtual" only in the context of "virtual reality" (3-D worlds). Use "online" only in the context of "online support groups". Complement or substitute product names with broader terms for the class of products (such as "mobile" or "smart phone" instead of "iphone"), especially if the application runs on different platforms. 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1a-i? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Primary Care through Electronic Medical Record Decision Support"

1a-ii) Non-web-based components or important co-interventions in title
Mention non-web-based components or important co-interventions in title, if any (e.g., "with telephone support").

Does your paper address subitem 1a-ii?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Primary Care through Electronic Medical Record Decision Support"

1a-iii) Primary condition or target group in the title
Mention primary condition or target group in the title, if any (e.g., "for children with Type I Diabetes") Example: A Web-based and Mobile Intervention with Telephone Support for Children with Type I Diabetes: Randomized Controlled Trial 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1a-iii? * Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Primary Care" 1b) ABSTRACT: Structured summary of trial design, methods, results, and conclusions NPT extension: Description of experimental treatment, comparator, care providers, centers, and blinding status.

1b-i) Key features/functionalities/components of the intervention and comparator in the METHODS section of the ABSTRACT
Mention key features/functionalities/components of the intervention and comparator in the abstract. If possible, also mention theories and principles used for designing the site. Keep in mind the needs of systematic reviewers and indexers by including important synonyms. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-i? * Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "RCT where 40 fully automated STOPP rules were implemented as EMR alerts during a 16-week intervention period. The control group did not receive the 40 STOPP rules (but would receive other alerts). "

1b-ii) Level of human involvement in the METHODS section of the ABSTRACT
Clarify the level of human involvement in the abstract, e.g., use phrases like "fully automated" vs. "therapist/nurse/care provider/physician-assisted" (mention number and expertise of providers involved, if any). (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem 1b-ii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "40 fully automated STOPP rules" 1b-iii) Open vs. closed, web-based (self-assessment) vs. face-to-face assessments in the METHODS section of the ABSTRACT Mention how participants were recruited (online vs. ofoine), e.g., from an open access website or from a clinic or a closed online user group (closed usergroup trial), and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment). Clearly say if outcomes were self-assessed through questionnaires (as common in web-based trials

Does your paper address subitem 1b-iii?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Participants were recruited through the OSCAR user group mailing list and in person user group meetings. "

1b-iv) RESULTS section in abstract must contain use data
Report number of participants enrolled/assessed in each group, the use/uptake of the intervention (e.g., attrition/adherence metrics, use over time, number of logins etc.), in addition to primary/secondary outcomes. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential Does your paper address subitem 1b-iv?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "28 physicians from 8 practices were recruited (16 in intervention, 12 in control groups)"

1b-v) CONCLUSIONS/DISCUSSION in abstract for negative trials
Conclusions/Discussions in abstract for negative trials: Discuss the primary outcome -if the trial is negative (primary outcome not changed), and the intervention was not used, discuss whether negative results are attributable to lack of uptake and discuss reasons. (Note: Only report in the abstract what the main paper is reporting. If this information is missing from the main body of text, consider adding it) 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem 1b-v?
Copy and paste relevant sections from the manuscript abstract (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "The calculated PIP rate was less than one quarter of the expected rate from the literature. Data quality probes highlighted issues related to completeness of data in areas of the EMR used for PIP reporting and by the decision support such as problem and medication lists. Users also highlighted areas for better integration of STOPP guidelines with prescribing workflows" 3b) Important changes to methods after trial commencement (such as eligibility criteria), with reasons Does your paper address CONSORT subitem 3b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study No changes to the protocol for this study.

3b-i) Bug /xes, Downtimes, Content Changes
Bug Uxes, Downtimes, Content Changes: ehealth systems are often dynamic systems. A description of changes to methods therefore also includes important changes made on the intervention or comparator during the trial (e.g., major bug Uxes or changes in the functionality or content) (5-iii) and other "unexpected events" that may have inouenced study design such as staff changes, system failures/downtimes, etc. [2].
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 3b-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study We have a section on data quality probes and the phase 2 reflection phase which look at potential adoption issues and these become a key focus on the paper.

4a) Eligibility criteria for participants
Does your paper address CONSORT subitem 4a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study

4a-i) Computer / Internet literacy
Computer / Internet literacy is often an implicit "de facto" eligibility criterion -this should be explicitly clariUed.

Does your paper address subitem 4a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study physicians "using the OSCAR EMR (v 12.x) for at least 12 months " 4a-ii) Open vs. closed, web-based vs. face-to-face assessments: Open vs. closed, web-based vs. face-to-face assessments: Mention how participants were recruited (online vs. ofoine), e.g., from an open access website or from a clinic, and clarify if this was a purely web-based trial, or there were face-to-face components (as part of the intervention or for assessment), i.e., to what degree got the study team to know the participant. In online-only trials, clarify if participants were quasi-anonymous and whether having multiple identities was possible or whether technical or logistical measures (e.g., cookies, email conUrmation, phone calls) were used to detect/prevent these.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4a-ii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Potential participants were screened by the PI on the phone or in person to ensure they met the criteria. "

4a-iii) Information giving during recruitment
Information given during recruitment. Specify how participants were briefed for recruitment and in the informed consent procedures (e.g., publish the informed consent documentation as appendix, see also item X26), as this information may have an effect on user self-selection, user expectation and may also bias results.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4a-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Through online and presentations "over 100 people were made aware of the study" 4b) Settings and locations where the data were collected Does your paper address CONSORT subitem 4b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Measurement of PIPs and DQ probes was completed using the UBC Department of Family Practice research network, which was developed with the Physicians Data Collaborative of BC. The network is based on hQuery, an open source tool, freely available on GitHub. The research network is designed to distribute querying of EMR health data without collection and storage of the patient level data. Only aggregate data (i.e. summary answers to queries) are collected in the central Hub (35). Patient privacy was maintained through the network as only practice level aggregate answers were returned through the network. "

4b-i) Report if outcomes were (self-)assessed through online questionnaires
Clearly report if outcomes were (self-)assessed through online questionnaires (as common in web-based trials) or otherwise.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 4b-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study The paper describes the methods of data collection 1. Self report survey (Demographics, etc) 2. Automated reporting from EMR 3. Focus Groups in phase 2 as part of the qualitative reflection

4b-ii) Report how institutional af/liations are displayed
Report how institutional afUliations are displayed to potential participants [on ehealth media], as afUliations with prestigious hospitals or universities may affect volunteer rates, use, and reactions with regards to an intervention.(Not a required itemdescribe only if this may bias results) 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem 4b-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Not described in the paper. The PI did comment on position as assistant professor at University of British Columbia and consents had UBC logo.

5)
The interventions for each group with sufUcient details to allow replication, including how and when they were actually administered

5-i) Mention names, credential, af/liations of the developers, sponsors, and owners
Mention names, credential, afUliations of the developers, sponsors, and owners [6] (if authors/evaluators are owners or developer of the software, this needs to be declared in a "Conoict of interest" section or mentioned elsewhere in the manuscript).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study The paper highlights the STOPP criteria and a wireframe illustration is provided to show how the intervention was displayed to the users.

5-ii) Describe the history/development process
Describe the history/development process of the application and previous formative evaluations (e.g., focus groups, usability testing), as these will have an impact on adoption/use rates and help with interpreting results.

Does your paper address subitem 5-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Not explicitly in the paper but references aspects from other publications on the development of the network.

5-iii) Revisions and updating
Revisions and updating. Clearly mention the date and/or version number of the application/intervention (and comparator, if applicable) evaluated, or describe whether the intervention underwent major changes during the evaluation process, or whether the development and/or content was "frozen" during the trial. Describe dynamic components such as news feeds or changing content which may have an impact on the replicability of the intervention (for unexpected events see item 3b).

Does your paper address subitem 5-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study OSCAR version 12 is described

5-iv) Quality assurance methods
Provide information on quality assurance methods to ensure accuracy and quality of information provided [1], if applicable.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-iv?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Several sub sections include reporting and discussion of the data quality probes 5-v) Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing ]owcharts of the algorithms used Ensure replicability by publishing the source code, and/or providing screenshots/screen-capture video, and/or providing oowcharts of the algorithms used. Replicability (i.e., other researchers should in principle be able to replicate the study) is a hallmark of scientiUc reporting.

Does your paper address subitem 5-v?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study All code, rules, etc are freely available.

5-vi) Digital preservation
Digital preservation: Provide the URL of the application, but as the intervention is likely to change or disappear over the course of the years; also make sure the intervention is archived (Internet Archive, webcitation.org, and/or publishing the source code or screenshots/videos alongside the article). As pages behind login screens cannot be archived, consider creating demo pages which are accessible without login.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-vi?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study This is part of the OSCAR EMR code or on GitHub

5-vii) Access
Access: Describe how participants accessed the application, in what setting/context, if they had to pay (or were paid) or not, whether they had to be a member of speciUc group. If known, describe how participants obtained "access to the platform and Internet" [1]. To ensure access for editors/reviewers/readers, consider to provide a "backdoor" login account or demo mode for reviewers/readers to explore the application (also important for archiving purposes, see vi).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-vii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "the guidelines were made freely available to the OSCAR EMR community after the study" subitem not at all important essential Does your paper address subitem 5-viii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study decision support embedded into EMR.

5-ix) Describe use parameters
Describe use parameters (e.g., intended "doses" and optimal timing for use). Clarify what instructions or recommendations were given to the user, e.g., regarding timing, frequency, heaviness of use, if any, or was the intervention used ad libitum.

Does your paper address subitem 5-ix?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study decision support alerts presented during patient encounters is described in the intervention subsection.

5-x) Clarify the level of human involvement
Clarify the level of human involvement (care providers or health professionals, also technical assistance) in the e-intervention or as co-intervention (detail number and expertise of professionals involved, if any, as well as "type of assistance offered, the timing and frequency of the support, how it is initiated, and the medium by which the assistance is delivered". It may be necessary to distinguish between the level of human involvement required for the trial, and the level of human involvement required for a routine application outside of a RCT setting (discuss under item 21generalizability).

Does your paper address subitem 5-x?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Implementation at a user level was required in the intervention group and facilitated by one of the authors (ID). Participants were instructed and walked through how to turn on the STOPP rules by trusting the STOPP content in the CDSS module. This downloaded and activated the STOPP rules for that user for the duration of the trial. "

5-xi) Report any prompts/reminders used
Report any prompts/reminders used: Clarify if there were prompts (letters, emails, phone calls, SMS) to use the application, what triggered them, frequency etc. It may be necessary to distinguish between the level of prompts/reminders required for the trial, and the level of prompts/reminders for a routine application outside of a RCT setting (discuss under item 21 -generalizability).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-xi? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study no prompts were used to remind people to use the intervention. The intervention, itself, was a set of prompts / alerts.

5-xii) Describe any co-interventions (incl. training/support)
Describe any co-interventions (incl. training/support): Clearly state any interventions that are provided in addition to the targeted eHealth intervention, as ehealth intervention may not be designed as stand-alone intervention. This includes training sessions and support [1]. It may be necessary to distinguish between the level of training required for the trial, and the level of training for a routine application outside of a RCT setting (discuss under item 21 -generalizability.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 5-xii? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study The only change was the addition of the 40 STOPP rules into the EMR's decision support system. 6a) Completely deUned pre-speciUed primary and secondary outcome measures, including how and when they were assessed Does your paper address CONSORT subitem 6a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes, the paper describes how baseline and postintervention queries were run to assess rates of potentially inappropriate prescriptions.

6a-i) Online questionnaires: describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed
If outcomes were obtained through online questionnaires, describe if they were validated for online use and apply CHERRIES items to describe how the questionnaires were designed/deployed [9].
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 6a-i?

Copy and paste relevant sections from manuscript text
Demographic questions were not validated prior to the study.

6a-ii) Describe whether and how "use" (including intensity of use/dosage) was de/ned/measured/monitored
Describe whether and how "use" (including intensity of use/dosage) was deUned/measured/monitored (logins, logUle analysis, etc.). Use/adoption metrics are important process outcomes that should be reported in any ehealth trial.

Does your paper address subitem 6a-ii?
Copy and paste relevant sections from manuscript text Rates of PIP was used as a proxy for measuring actual displaying of the alerts -discussion goes into detail about the need for deeper measuring of specific workflows, which were not possible in the current iteration of the OSCAR EMR.

6a-iii) Describe whether, how, and when qualitative feedback from participants was obtained
Describe whether, how, and when qualitative feedback from participants was obtained (e.g., through emails, feedback forms, interviews, focus groups).

2 3 4 5
subitem not at all important essential

Does your paper address subitem 6a-iii?
Copy and paste relevant sections from manuscript text The study was designed with a phase 2 for qualitative feedback. There are several sub-sections on phase 2.
6b) Any changes to trial outcomes after the trial commenced, with reasons Does your paper address CONSORT subitem 6b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study No changes 7a) How sample size was determined NPT: When applicable, details of whether and how the clustering by care provides or centers was addressed

7a-i) Describe whether and how expected attrition was taken into account when calculating the sample size
Describe whether and how expected attrition was taken into account when calculating the sample size.

Does your paper address subitem 7a-i?
Copy and paste relevant sections from manuscript title (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Sample Size: Sample size was calculated assuming a PIP rate of 20% (10,27,34) and an expected relative reduction of 20% in PIPs (absolute reduction of 4%). Using a power of 0.8 and alpha = 0.05, estimating two physicians may be lost to follow up, we predicted the need for 12 physician practices in each arm and 900 encounters with patients 65+." 7b) When applicable, explanation of any interim analyses and stopping guidelines Does your paper address CONSORT subitem 7b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study N/A 8a) Method used to generate the random allocation sequence NPT: When applicable, how care providers were allocated to each trial group Does your paper address CONSORT subitem 8a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Clinics were randomly assigned (equal distribution by clinic) to intervention or control using a random number generator. Randomization was stratified into small (<4 MDs) and large clinics (>=4 MDs)."

8b) Type of randomisation; details of any restriction (such as blocking and block size)
Does your paper address CONSORT subitem 8b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Randomization was stratified into small (<4 MDs) and large clinics (>=4 MDs)." 9) Mechanism used to implement the random allocation sequence (such as sequentially numbered containers), describing any steps taken to conceal the sequence until interventions were assigned Does your paper address CONSORT subitem 9? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Clinics were randomly assigned (equal distribution by clinic) to intervention or control using a random number generator. Randomization was stratified into small (<4 MDs) and large clinics (>=4 MDs)." This was done with excel.
10) Who generated the random allocation sequence, who enrolled participants, and who assigned participants to interventions Does your paper address CONSORT subitem 10? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "random number generator generating the list prior to the study recruitment and allocation" 11a) If done, who was blinded after assignment to interventions (for example, participants, care providers, those assessing outcomes) and how NPT: Whether or not administering co-interventions were blinded to group assignment

11a-i) Specify who was blinded, and who wasn't
Specify who was blinded, and who wasn't. Usually, in web-based trials it is not possible to blind the participants [1, 3] (this should be clearly acknowledged), but it may be possible to blind outcome assessors, those doing data analysis or those administering co-interventions (if any).

2 3 4 5
subitem not at all important essential Does your paper address subitem 11a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study no blinding in this study.
11a-ii) Discuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator" Informed consent procedures (4a-ii) can create biases and certain expectationsdiscuss e.g., whether participants knew which intervention was the "intervention of interest" and which one was the "comparator".

Does your paper address subitem 11a-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Participants were aware of the STOPP criteria but were given no training / review of what these criteria were in the study.

11b) If relevant, description of the similarity of interventions
(this item is usually not relevant for ehealth trials as it refers to similarity of a placebo or sham intervention to a active medication/intervention) Does your paper address CONSORT subitem 11b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study NA 12a) Statistical methods used to compare groups for primary and secondary outcomes NPT: When applicable, details of whether and how the clustering by care providers or centers was addressed Does your paper address CONSORT subitem 12a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "PIPs for each of the STOPP rules and DQ probes were measured for each clinic. Statistical regression models which account for the clustered nature of the data were fit to assess the primary outcome (geeglm in R package gee pack)"

12a-i) Imputation techniques to deal with attrition / missing values
Imputation techniques to deal with attrition / missing values: Not all participants will use the intervention/comparator as intended and attrition is typically high in ehealth trials. Specify how participants who did not use the application or dropped out from the trial were treated in the statistical analysis (a complete case analysis is strongly discouraged, and simple imputation techniques such as LOCF may also be problematic [4]).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 12a-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Intention to treat analysis. No users dropped out of the study.
12b) Methods for additional analyses, such as subgroup analyses and adjusted analyses Does your paper address CONSORT subitem 12b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study

X26) REB/IRB Approval and Ethical
Considerations [recommended as subheading under "Methods"] (not a CONSORT item)

X26-i) Comment on ethics committee approval
1 2 3 4 5 subitem not at all important essential

Does your paper address subitem X26-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study

Methods concludes with:
"This study was registered at clinical trials.gov (NCT02130895) and received clinical ethics approval from the University of British Columbia (H14-00797)."

x26-ii) Outline informed consent procedures
Outline informed consent procedures e.g., if consent was obtained ofoine or online (how? Checkbox, etc.?), and what information was provided (see 4a-ii). See [6] for some items to be included in informed consent documents.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem X26-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "Twenty-eight physicians across eight practices consented in person "

X26-iii) Safety and security procedures
Safety and security procedures, incl. privacy considerations, and any steps taken to reduce the likelihood or detection of harm (e.g., education and training, availability of a hotline) 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem X26-iii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Not highlighted in the paper -participants were invited to connect with the PI or their EMR service provider in case of any concerns.
"One participant reported a technical problem with their EMR that was thought to be related to the intervention and later discovered to be unrelated. "

RESULTS
13a) For each group, the numbers of participants who were randomly assigned, received intended treatment, and were analysed for the primary outcome NPT: The number of care providers or centers performing the intervention in each group and the number of patients treated by each care provider in each center Does your paper address CONSORT subitem 13a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes -highlighted that this was intention to treat with no change or loss to follow up.
13b) For each group, losses and exclusions after randomisation, together with reasons Does your paper address CONSORT subitem 13b? (NOTE: Preferably, this is shown in a CONSORT ]ow diagram) * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study "None were lost to follow up ( Figure 2)"

13b-i) Attrition diagram
Strongly recommended: An attrition diagram (e.g., proportion of participants still logging in or using the intervention/comparator in each group plotted over time, similar to a survival curve) or other Ugures or tables demonstrating usage/dose/engagement.

2 3 4 5
subitem not at all important essential

Does your paper address subitem 13b-i?
Copy and paste relevant sections from the manuscript or cite the Ugure number if applicable (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study See Figure 2 14a) Dates deUning the periods of recruitment and follow-up Does your paper address CONSORT subitem 14a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes -dates for presentations as well as dates for the study were included: "The study was completed from February-October, 2015. "

14a-i) Indicate if critical "secular events" fell into the study period
Indicate if critical "secular events" fell into the study period, e.g., signiUcant changes in Internet resources available or "changes in computer hardware or Internet delivery resources" 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem 14a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study There were no secular events that we were aware of.
14b) Why the trial ended or was stopped (early) Does your paper address CONSORT subitem 14b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study

) and centers (volume) in each group
Does your paper address CONSORT subitem 15? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Table 1 highlights some of the aspects as well as information in the prose -these characteristics are related to PIPs

15-i) Report demographics associated with digital divide issues
In ehealth trials it is particularly important to report demographics associated with digital divide issues, such as age, education, gender, social-economic status, computer/Internet/ehealth literacy of the participants, if known.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 15-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study This study is focused on those already using an EMR as one of the inclusion criteria.
16) For each group, number of participants (denominator) included in each analysis and whether the analysis was by original assigned groups

16-i) Report multiple "denominators" and provide de/nitions
Report multiple "denominators" and provide deUnitions: Report N's (and effect sizes) "across a range of study participation [and use] thresholds" [1], e.g., N exposed, N consented, N used more than x times, N used more than y weeks, N participants "used" the intervention/comparator at speciUc pre-deUned time points of interest (in absolute and relative numbers per group). Always clearly deUne "use" of the intervention.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 16-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study yes -intention to treat analysis was performed.

16-ii) Primary analysis should be intent-to-treat
Primary analysis should be intent-to-treat, secondary analyses could include comparing only "users", with the appropriate caveats that this is no longer a randomized sample (see 18-i).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 16-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study yes.
17a) For each primary and secondary outcome, results for each group, and the estimated effect size and its precision (such as 95% conUdence interval) Does your paper address CONSORT subitem 17a? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study We describe the analysis of the primary outcome with a P value (0.8)

17a-i) Presentation of process outcomes such as metrics of use and intensity of use
In addition to primary/secondary (clinical) outcomes, the presentation of process outcomes such as metrics of use and intensity of use (dose, exposure) and their operational deUnitions is critical. This does not only refer to metrics of attrition (13-b) (often a binary variable), but also to more continuous exposure metrics such as "average session length". These must be accompanied by a technical description how a metric like a "session" is deUned (e.g., timeout after idle time) [1] (report under item 6a).
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 17a-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study We used data quality probes to further assess the system 17b) For binary outcomes, presentation of both absolute and relative effect sizes is recommended Does your paper address CONSORT subitem 17b? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study NA 18) Results of any other analyses performed, including subgroup analyses and adjusted analyses, distinguishing pre-speciUed from exploratory Does your paper address CONSORT subitem 18? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study NA

18-i) Subgroup analysis of comparing only users
A subgroup analysis of comparing only users is not uncommon in ehealth trials, but if done, it must be stressed that this is a self-selected sample and no longer an unbiased sample from a randomized trial (see 16-iii).

Does your paper address subitem 18-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study NA 19) All important harms or unintended effects in each group (for speciUc guidance see CONSORT for harms)

Does your paper address CONSORT subitem 19? *
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study The unintended effect would be an measurable increase in inappropriate prescriptions, which was not seen

19-i) Include privacy breaches, technical problems
Include privacy breaches, technical problems. This does not only include physical "harm" to participants, but also incidents such as perceived or real privacy breaches [1], technical problems, and other unexpected/unintended incidents. "Unintended effects" also includes unintended positive effects [2].

Does your paper address subitem 19-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study We described the implementation and referenced design papers on the network. The technical implementation was surprisingly smooth.

19-ii) Include qualitative feedback from participants or observations from staff/researchers
Include qualitative feedback from participants or observations from staff/researchers, if available, on strengths and shortcomings of the application, especially if they point to unintended/unexpected effects or uses. This includes (if available) reasons for why people did or did not use the application as intended by the developers.

Does your paper address subitem 19-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Phase 2 focuses on this aspect explicitly in the paper.

22)
Interpretation consistent with results, balancing beneUts and harms, and considering other relevant evidence NPT: In addition, take into account the choice of the comparator, lack of or partial blinding, and unequal expertise of care providers or centers in each group

22-i) Restate study questions and summarize the answers suggested by the data, starting with primary outcomes and process outcomes (use)
Does your paper address subitem 22-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes: "We sought to understand: How can an existing clinical decision support tool implement a complex set of evidence-based rules into primary care clinical practice and how does this impact prescribing?"

22-ii) Highlight unanswered new questions, suggest future research
Highlight unanswered new questions, suggest future research.

Does your paper address subitem 22-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study We have a future direction subsection. 20) Trial limitations, addressing sources of potential bias, imprecision, and, if relevant, multiplicity of analyses

20-i) Typical limitations in ehealth trials
Typical limitations in ehealth trials: Participants in ehealth trials are rarely blinded.
Ehealth trials often look at a multiplicity of outcomes, increasing risk for a Type I error. Discuss biases due to non-use of the intervention/usability issues, biases through informed consent procedures, unexpected events.
1 2 3 4 5 subitem not at all important essential Does your paper address subitem 20-i? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study subsection in the discussion exploring limitations 21) Generalisability (external validity, applicability) of the trial Undings NPT: External validity of the trial Undings according to the intervention, comparators, patients, and care providers or centers involved in the trial

21-i) Generalizability to other populations
Generalizability to other populations: In particular, discuss generalizability to a general Internet population, outside of a RCT setting, and general patient population, including applicability of the study results for other organizations 1 2 3 4 5 subitem not at all important essential

Does your paper address subitem 21-i?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes. This is a pragmatic trial intentionally recruiting community family physicians (vs academic centres)

21-ii) Discuss if there were elements in the RCT that would be different in a routine application setting
Discuss if there were elements in the RCT that would be different in a routine application setting (e.g., prompts/reminders, more human involvement, training sessions or other co-interventions) and what impact the omission of these elements could have on use, adoption, or outcomes if the intervention is applied outside of a RCT setting.

Does your paper address subitem 21-ii?
Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study routine session was selected for this trial intentionally to understand the potential benefits and challenges of implementing these rules broadly in the community.

23) Registration number and name of trial registry
Does your paper address CONSORT subitem 23? * Copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study Yes: "This study was registered at clinical trials.gov (NCT02130895) and received clinical ethics approval from the University of British Columbia (H14-00797)." 24) Where the full trial protocol can be accessed, if available Does your paper address CONSORT subitem 24? * Cite a Multimedia Appendix, other reference, or copy and paste relevant sections from the manuscript (include quotes in quotation marks "like this" to indicate direct quotes from your manuscript), or elaborate on this item by providing additional information not in the ms, or brieoy explain why the item is not applicable/relevant for your study ClinicalTrials.gov 25) Sources of funding and other support (such as supply of drugs), role of funders Does your paper address CONSORT subitem 25? *