Geisinger is an integrated health care delivery system located in central and northeastern Pennsylvania, serving more than 1 million patients yearly. Geisinger maintains 44 primary care and over 100 specialty clinics, which include 8 neurology practices with 43 neurology physicians all using the same EHR system.

We conducted a prospective stratified cluster-randomized controlled trial across 38 primary care sites to assess the impact of an interruptive electronic alert on the management of primary headache disorders. Eligible patients were enrolled and followed for 6 months post index encounter to evaluate headache management-related outcomes. It was uploaded to ClinicalTrials.gov and can be found using the registration number NCT05067725. The trial followed CONSORT (Consolidated Standards of Reporting Trials) reporting guidelines for randomized trials. We have also reported the e-CONSORT per the journal requirements [15].

In total, 38 of 44 primary care clinic sites were selected for randomization based on their ability to participate. The sites were first stratified into 9 possible blocks based on 2 criteria: geographical location (west, central, or northeast) and patient-to–health care provider ratio (low, moderate, or high). Note that 3 of the 9 possible blocks did not contain any sites, leaving only 6 blocks for this study. Within each block, simple randomization was then used to assign each site to either the intervention or control arm, using a 1:1 ratio. A Microsoft Excel 365 (Microsoft Corporation) random number generator was used to generate the simple randomization allocation sequence. Patients received the treatment to which their site had been assigned, making this a cluster-randomized study with sites as clusters (Multimedia Appendix 1).

Patients were eligible if (1) the criteria for the alert were met, which included an in-person or telemedicine encounter and a headache diagnosis or chief complaint of headache (Multimedia Appendix 2); (2) they were aged 18 years or older; and (3) postvisit criteria were met, which included a confirmed diagnosis of primary headache disorder by manual medical record review and a minimum baseline Headache Impact Test 6 (HIT-6) score of 50 or higher or headache frequency of 12 days or greater in the last 3 months. Clinical input and the scale guidance document were used to determine thresholds for HIT-6 scores and headache frequency [16]. Manual medical record reviews were conducted by study team members (HA, AP, and SP) under the guidance of the neurologist (SF) to confirm headache diagnosis. Types of headaches in the analytic sample included migraine, chronic daily, tension-type, cluster, or medication overuse headaches, collectively grouped as “headaches.” Patients were excluded if they had a serious systemic illness (eg, headaches due to hypertensive emergencies, hepatic or renal failure, or cardiac failure), secondary headaches (eg, concussion), were pregnant, or were actively followed with neurology [17]. Patients with serious systemic illness were excluded from our study as it would have been difficult to distinguish whether headaches observed among them were due to primary or secondary factors.

Within 2 weeks after their scheduled visit, eligible patients were contacted by the Survey Research and Recruitment Core to collect baseline measures (which determined minimum headache eligibility) and to gain verbal consent. Patient contact was attempted 7 times. Figure 1 represents this study’s process of patient selection, enrollment, and follow-up.

A clinician-facing interruptive EHR-based alert in the form of a Best Practice Advisory was built based on input from an expert panel of neurologists, primary care providers (PCPs), pharmacists, and informaticians. PCPs include primary care physicians and advanced practitioners such as nurse practitioners and physician assistants. The first iteration of the alert was developed based on input received from the needs assessment conducted with the PCPs. This version was piloted at 2 primary care sites over 3 months [18]. Face-to-face education of PCPs at the pilot sites was undertaken to familiarize health care providers with the utility of the alert [18]. A simple pre-post assessment of the 900 eligible patient encounters during this pilot phase demonstrated a significant reduction in neurology referrals post implementation of the electronic alert. Following the pilot implementation, feedback was collected from 10 PCPs at these sites about the structure, verbiage, and utility of the electronic alert. Subsequent modifications were made based on this feedback before full implementation.

The Best Practice Advisory consisted of four parts: (1) a questionnaire to assist the PCP in characterizing the patient’s headache (Multimedia Appendix 3); (2) a “SmartSet” guide for ordering medications, imaging, laboratories, and referrals using evidence-based guidelines (Multimedia Appendix 3); (3) a hyperlink to allow for electronic consultation with a neurologist; and (4) a link to the “synopsis” which provided a summary of prior patient headache care. The “SmartSet” guide could be accessed directly within the alert by clicking on the “Open Express Lane” button to facilitate quick ordering for health care providers. The alert appeared within the EHR when a PCP entered a headache diagnosis for an in-person or telemedicine encounter. All actions taken in the alert were recorded, although health care providers could dismiss them without any action. Suppression criteria were built to lock out and avoid repeated firing of the alert. The alert was suppressed for 24 hours if the dismissal reason was headache-controlled and for 365 days for PCP long-term management. The alert could also be dismissed by acknowledging that the health care provider was previewing the patients’ medical record, in this case, the alert would refire during the same encounter.

An invisible or silent alert was activated at control sites that would be triggered by identical criteria to the intervention arm, however, this alert was not visible to the clinicians, so no suppression criteria were developed for control sites. The silent alert was developed to help generate a list of eligible patients for the control group that would be comparable to those identified in the intervention arm.

Before full implementation, the primary care leadership team was encouraged to disseminate information about the tool to PCPs in the intervention cohort through local departmental meetings and direct emails, including text and short video instructions.

Data sources included the EHR and survey results from patients at baseline, 3 months, and 6 months. The surveys conducted by the Survey Research and Recruitment Core included 6 questions from the HIT-6 questionnaire, which measures headache impact on a participant’s ability to function in day-to-day life [19]; 1 question on pain intensity; 1 on the frequency of headaches; 3 from the Migraine Treatment Optimization Questionnaire (M-TOQ), which determines if a medication treatment for a participant is optimal; and 2 asking for sociodemographic information that could not be pulled from the EHR [20]. The M-TOQ questions were modified for our study by replacing the word “migraine” with “headache.” Prior to study enrollment, the survey was piloted among a sample (n=10) of patients from pilot sites to ensure appropriateness and check for face validity. Patients had the option to opt-out at any time during this study. See Multimedia Appendix 4 for the full survey guide.

A minimum difference of equal to or greater than a 2.3-point (SD 4.3) reduction on the HIT-6 scale represents a clinically important and significant change [22]. We needed to enroll at least a total of 66 patients per arm to detect this change with 80% power at a significance level of .05 and an intraclass correlation of 0.02. The sample size was calculated using an online calculator developed by the University of California San Francisco [24].

The total number of patients in each arm was counted to ensure a balanced distribution of patient numbers. Descriptive statistics were computed for continuous variables as means or medians with SD or IQRs, and as frequency and percentages for categorical variables when appropriate for baseline patient-reported outcomes and socioeconomic and demographic characteristics (ie, sex, age, race, ethnicity, insurance, education, and income). Appropriate statistical tests were used to compare differences in these characteristics between the 2 groups and within groups: 2-tailed independent t tests or paired t tests were used for continuous data, χ² for categorical data, and Mann-Whitney for nonnormal categorical or continuous data.

We constructed GEE for all outcome comparisons between groups, adjusting for effects of treatment, phase (baseline, 6 months), treatment-by-phase interaction, and demographics while accounting for clustering of patients per primary care site. The GEE model with log link was used, considering the skewness of data observed for headache days and pain scores (Multimedia Appendix 5). For all the binary variables, a log link with binomial distribution was used, while an identity link with normal distribution was used for HIT-6 scores. As a first step, we performed an unadjusted GEE model with this study’s groups as only a univariate analysis. Multivariable (adjusted) models only included demographic variables that were significantly different between groups (P<.05). For the GEE analysis, we reported odds ratios or estimates and associated CIs. All analyses were conducted using SAS Enterprise Guide software (version 8.3 for Windows; SAS Institute Inc).

The analysis was conducted based on the principle of “intention-to-treat (ITT)” analysis. All patients who consented and met the inclusion criteria were included in the analysis per the group they were recruited into. The analysis was performed to assess the difference in outcomes at the individual patient level. For missing data via survey collection, we applied a last observation carried forward (LOCF) technique to maximize the number of observations available for analysis [25]. The results from the LOCF technique were further compared to those from the measured data to assess the impact of loss to follow-up.

This study was an evaluation of a quality improvement initiative that was undertaken at Geisinger with the intent to affect clinical practice. As a result, it was determined as “not human subjects research” by Geisinger’s institutional review board (2021‐0729).

The alert was fired for a total of 9239 times in 3183 patients (intervention arm: 2677 times in 1507 patients; control arm: 6562 times in 1676 patients) between December 2021 and February 2022. We excluded 1828 patients for not meeting eligibility criteria. We manually verified the diagnosis of primary headache among 729 of the remaining 1355 patients who were eligible for consent and baseline survey assessment. We obtained verbal consent from 221 patients at baseline but only enrolled the 203 patients that either had an HIT-6 score of ≥50 or a headache frequency of >12 days, as per our inclusion criteria (Figure 1). At the end of 6 months, follow-up patient-reported data were available for 88/203 (43.3%) patients.

All baseline characteristics were similar between groups for patients enrolled in this study except for mean age, which was higher in the intervention group (mean 43, SD 14.4 y vs mean 39, SD 14.4 y, P=.04; Table 1). There was no difference in the HIT-6 scores, headache frequency and intensity, or patient-reported use of medications at baseline between groups (HIT-6 scores: mean 63, SD 6.9 vs mean 61.8, SD 6.6; headache days in the past 3 months: median 20.5, IQR 10.0 - 45.0 vs median 20, IQR 10.0 - 45.0; pain scores: median 7, IQR 6.0 - 8.0 vs median 7, IQR 5.0 - 8.0; reported use of preventive or abortive medications: 63/98, 64% vs 57/105, 54.2%). The majority of the patients were female, were White, were non-Hispanic, had a commercial insurance plan, had an educational qualification of greater than high school, and had an annual household income between US $50,000 to US $100,000. Similarly, there was no statistically significant difference in the baseline characteristics observed between groups for patients that completed the 6-month follow-up.

HIT-6 scores improved significantly from baseline in the intervention arm at 6 months (3.5 points; P=.009) but not in the control group (1.40 points; P=.23); additionally, the improvements from baseline did not differ significantly between groups (Table 2). The intracluster correlation coefficient observed was 0.411 for HIT-6 scores.

Change in the number of headache days and pain score did not differ between groups (Table 2). Similarly, compared to baseline, there was no difference in the proportion of patients being referred to neurology, using ED for all-cause or headache-specific reasons, or initiating new abortive or preventative treatment in the 6 months post intervention (Table 3).

Results from the LOCF models for HIT-6 scores, headache days, and pain intensity are listed in Table 4. Findings from the LOCF models were similar to those from the base models presented in Table 2. It was observed that there was a similar significant improvement in HIT-6 scores in the intervention arm, and a reduction in the headache days in the control arm from baseline to 6 months, however, the difference between groups was not statistically significant.

Of the 2677 alert firings in the intervention arm, 2228 (83.23%) were overridden or dismissed, 146 (5.45%) were ignored, and 303 (11.32%) action was taken. Of the 205 PCPs exposed to the intervention alert, 15 provided feedback, 3 found the alert helpful, and 12 expressed dissatisfaction due to alert firing on inappropriate patients or noting the alert was disruptive to workflow. A manual medical record review of 1355 patients revealed that 729 (53.80%) had a confirmed diagnosis of primary headache disorder. The remaining 626 (46.20%) either did not have a confirmed diagnosis of primary headache disorder or were already followed by neurology for headaches.