Computerized clinical decision support systems (CDSSs) can be defined as the information and communication systems that provide clinicians or patients with timely, accurate, and appropriate knowledge to enhance patient care [1]. They have emerged as an attractive approach to improving compliance of clinicians with clinical practice guidelines (CPGs). Their potential to improve knowledge translation of the CPGs is being considered for an increasing variety of clinical contexts. However, their effectiveness in achieving this continues to be controversial.

Evidence describing the effectiveness of CDSSs in improving knowledge translation comes from a number of systematic reviews, including studies carried out in a variety of contexts [2-5]. The majority of studies reviewed were conducted in outpatient and academic settings.

The effectiveness of CDSSs in these settings is supported through improvements in process adherence, including medication ordering, vaccinations, test ordering, and diagnosis and disease management [3-5]. However, not all clinical processes have shown improvements after implementing a CDSS [3,4]. Of note is that most of the research has been related to diabetes and other cardiovascular risks, areas that have clear diagnostic criteria and well-supported management strategies and disease monitoring processes.

Pain management, on the other hand, has less clarity for many diagnostic criteria and less consensus on effective treatment and monitoring. Thus, the knowledge base is more difficult to translate. Little research has been conducted on the effect of CDSS on pain management generally, and neuropathic pain more specifically. The available literature, including 1 systematic review, is primarily focused on cancer pain management [6-11]. To a lesser extent, CDSS pain research has been conducted around chronic noncancer pain, including headaches, as well as low back pain (LBP) and neuropathic pain [6,12]. A systematic review emphasized the need for further research in this area as all studies included were nonexperimental [6]. Although a more recent experimental study examined the impact of a CDSS on the outcomes of patients with chronic pain in primary care, results demonstrated that these patients were still undertreated or inadequately treated [12].

Factors that limit and facilitate the effectiveness of CDSS have been reported. Most studies are able to demonstrate improvements in clinical process, but very little data provide information about improved patient outcomes [6]. Many patient-specific outcomes for chronic pain have not been explored, including health care utilization, health care costs, pain relief, pain medication usage, communication with providers, functional status, and quality of life. Barriers to effectiveness include inadequate training, poor usability or integration into practice workflow, and nonacceptance by practitioners of computerized recommendations [13-15]. Success has been more common in systems that prompt users to use the tools, those that have been developed by trial authors rather than externally and those that provide recommendations and not only assessments [2].

The Veterans Affairs (VA) in the United States developed a CDSS for neuropathic pain in their national electronic medical record (EMR) that involved significant clinician engagement in prototype development to improve use [16]. Although the significant clinician engagement resulted in improvements in the focus, scope, content, and presentation of the CDSS, the effects on patient outcomes were not evaluated [16]. In addition, the VA neuropathic pain CDSS is no longer active as frequent updates were required to maintain current clinical knowledge, which was not possible once the research project ended [17].

Neuropathic pain is a unique subset of pain conditions that often becomes chronic, decreasing function and quality of life [18,19]. It can be difficult to diagnose even though it has specific diagnostic criteria. Treatment is also unique in that pain responds best to medications that are used mainly to treat seizures or to treat depression. It is defined by the International Association for the Study of Pain as “pain caused by a lesion or a disease of the somatosensory system” [20]. The estimated prevalence of neuropathic pain in the general population is 2% to 3%; however, there are estimates that 7% to 8% of the population experience pain with neuropathic components [20].

Overall, the challenge of diagnosing and treating patients with neuropathic pain often leads to tension in clinician-patient relationships, frequent and prolonged visits with high emotional intensity, poor patient compliance, poor clinical outcomes, and sometimes refusal of access to care for those who identify as having chronic pain. It is clear that support is needed to facilitate optimal care for patients with chronic pain in primary care [21].

Our aim was to improve the adherence to CPG recommendations in primary care for the diagnosis and treatment of neuropathic pain through CDSS. The CDSS development has been reported previously [22]. Our hypothesis was that improved management would result from a tool with high usability combined with recommendations that were acceptable to clinicians. We sought to answer the following research questions: (1) How does self-reported awareness of, agreement with, and adoption of key CPG recommendations for neuropathic pain change among clinicians during the year following the introduction of the CDSS? and (2) How does the observed adherence to key CPG recommendations for neuropathic pain change from 1 year preceding to 1 year following the introduction of the CDSS?

The CDSS and the evaluation study were both developed through engagement of an advisory board made up of clinician researchers, software developers, end users, CPG developers, and information systems experts. The project was based at the Department of Family Medicine at McMaster University in Hamilton, Canada. The CDSS was designed for the Open Source Clinical Application Resource (OSCAR) EMR [23]. At the time this project was launched, an overhaul of the user interface for OSCAR was in process, presenting the opportunity for the function of the CDSS within the EMR to be optimized.

Overall, 3 academic family medicine clinics, all involved in delivering McMaster’s family medicine residency training program, were the sites where development, testing, implementation, and evaluation took place. Together, these clinics serve 40,000 patients, with 169 clinicians, including family physicians, family medicine residents, and nurse practitioners.

The study was approved by the Hamilton Integrated Research Ethics Board, reference number 13-136.

We began with a broad consultation with the varied stakeholders on the advisory board, as described above. This was followed with focus group discussions with interprofessional end users to determine the requirements for the CDSS content, appearance, function, and workflow. Practice recommendations were drawn from the Canadian guidelines for management of neuropathic pain [24]. The consultation process informed the development approach outlined below. A prototype was developed and followed with iterative cycles of usability testing and modification. This process has been reported previously [22].

The final version of the CDSS was created and integrated into the OSCAR EMR code. Its functionality was designed to be optimal with a new user interface that was scheduled to be in use by the time of the study. However, this did not occur and the CDSS was deployed within the older user interface environment. Although all functions were available on the older user interface, gaining access to and the appearance of the CDSS were not as clear or easy to follow as the prototype design.

Adherence is defined as objective, observed practice of the key guideline recommendations. This was measured through a visual review of the EMR records. As neuropathic pain is a relatively rare condition in primary care, and our goal was to audit a large number of clinical encounters that involved pain management, we elected to select records based on a query for clinical encounters involving pain, rather than based on the clinicians who were providing care. As the introduction of the CDSS could have an impact on all clinicians, not only those who were enrolled in the study, all patients in the clinic and their encounters with all clinicians were deemed eligible to be sampled.

We selected a sample of 100 patients for chart review before CDSS introduction and an independent sample of another 100 patients for chart review at 12 months following CDSS introduction. We used the following selection procedure. All patients in the clinic aged over 17 years with a clinic visit in the past 12 months were deemed eligible. Queries were developed to identify patients with new acute neuropathic pain or an acute exacerbation of neuropathic pain. From the list generated from the EMR, 100 patients were randomly selected for review. This same procedure was conducted at 12 months following CDSS introduction, with the removal of any patients who had been assessed already in the first selection and appearing again in the second selection.

The review of records was conducted by an independent medical doctor who was not an investigator. The reviewer used a data template to extract pertinent measures. For the first 10 patients, records were independently reviewed by one of the investigators (DG) to determine that interrater reliability reached a kappa of 0.95. All clinical encounters experienced by any clinician were reviewed.

We defined CDSS utilization as the entering and saving of data in any of the fields of any of the forms of the CDSS. Opening the form, viewing it, or entering data that were not saved was not captured in our utilization query. A query of the data saved in the CDSS was run at the 6- and 12-month time points. Provider names were recoded with study identifiers, as well as provider type. Descriptive statistics were used to analyze the patterns of utilization.

A total of 5 focus groups were conducted with 2 at each of the 2 clinics and 1 at the third. All clinicians who had enrolled in the study were eligible to participate in a focus group, with the purpose of assessing satisfaction with, and overall experience of, using the CDSS. Clinicians were welcome to participate in the focus group discussion if they felt familiar enough with the CDSS to comment on their own experience. There were at least 2 research team members attending each group, and all were led by an experienced facilitator (KN). Group discussions were digitally recorded and transcribed verbatim. Overall, 2 team members (MA and KN) completed the coding and analysis of the data.

Participants are described in Table 1. There was a population of 169 available clinicians among the 3 study clinics at baseline. Those who agreed to participate in the study by completing questionnaires included 34 family physicians, 75 residents (first year, n=64; second year, n=11), and 9 nurse practitioners, for a total of 118 of 169 clinicians, or 69.8% of available clinicians at baseline for all 3 sites.

Of the 118 clinicians who consented to participate in the study, 100 (84.7%) completed the baseline and 66 (55.9%) completed the 6-month questionnaires. There were fewer eligible participants for the 12-month time point owing to 1 clinic being delayed in starting and the second-year residents graduating before study completion. This allowed only 2 time points to be collected for those participants. Thus, there were 86 eligible participants and 35 (40%) completed questionnaires at 12 months.

Clinician awareness, agreement, and adoption of guideline recommendations was high at baseline in this study population, with mean scores in the top quartile for all parameters. Scores at 6 and 12 months when compared with baseline were, however, significantly higher for both awareness and adoption.

Table 2 describes the characteristics of the patient sample populations selected to audit the adherence of clinicians to guideline recommendations before and after the introduction of the CDSS. Characteristics of pre- and postsamples were similar. The majority of the sample was female.

Table 2 also reports the degree of adherence to history, examination, and treatment recommendations for neuropathic pain. As this is a primary care population, many visits were not related to a complaint of pain, but for about half of all the visits, pain was a concern. Among the visits where pain was a concern, a significantly higher number of visits in the post-CDSS audit (50.9% [171/336] post vs 39.0% [156/400] pre; P=.001) described features of neuropathic pain as being either present or absent. Similarly, there was a significantly higher number of visits for which sensation testing was carried out and recorded during the post-CDSS period (35.1% post vs 12.3% pre; P<.001).

Finally, the use of first-line medication was higher than the use of second-line medication, even at baseline. There was no significant change in the initiation of either first-line or second-line medications from pre-CDSS to post-CDSS periods.

When analyzing by patient, rather than the encounters, we found that 56 of the 100 patients in the pre-CDSS sample and 69 of the 100 patients in the post-CDSS sample were either newly prescribed or already taking a first-line neuropathic pain medication. Although this suggests an increase in appropriate prescribing over the study year, the difference did not reach statistical significance, at P=.06.

At the 12-month time point, 18 of 169 possible clinicians (10.7%) had saved data in the CDSS neuropathic pain forms. A total of 1352 fields were saved on 40 neuropathic pain forms. Utilization was highest among nurse practitioners, with an average of 61 forms saved per nurse, 12 forms per resident, and 5.4 forms per physician.

Among study participants, both those who used any of the CDSS forms and those who did not, showed significant increases in awareness and adoption of guideline recommendations. Users and nonusers had similar awareness and adoption at baseline.

Among the 5 focus groups at 3 sites, there were 23 participants, including 10 physicians, 10 residents, and 3 nurse practitioners. Participants had all been introduced to the CDSS either through in-person or online training or through contact with colleague champions. Several key themes were evident.

Access and workflow were commented on most frequently. Owing to the combination of the newer format of CDSS and the older format of user interface, people had difficulty finding the CDSS link. In addition, its integration with other chronic disease management tools was an unfamiliar feature, and they did not completely trust its function. As one participant reported, “...I thought it was a great idea and I was enthusiastic...if it takes more than 3 seconds or something like that, it quickly falls off your priority list to do.”

The format and function of the CDSS itself was appealing once they had this open. However, some found that there were more information and data fields available than what they wished to make use of. Some felt that the number of patients they managed with pain was too low to develop ease with using the CDSS. A participant reported, “I remember going in there and clicking around and finding all the different things that were in there and I think if I had spent more time in there and used it, it probably would have been valuable.”

Most had opened and referred to the CDSS for the guideline recommendations 1 or more times, even if they had not entered data or used the dynamic functions. Reference information related to guideline recommendations was generally valuable and well accessed. Many participants had opened the CDSS to view the material there, often to confirm that their practice was fitting with guidelines. This was particularly common for first year residents and for nurse practitioners. As one resident said, “...when I have read it through enough times it gave me that practice and just, just asking those questions. So, I use it more as a reference tool I think.”

Our priority was to create a CDSS that was appealing to clinicians and therefore would be used in a way that would translate knowledge into practice. We also discovered that clinician knowledge concerning guideline recommendations for neuropathic pain was higher in our study participants at baseline than anticipated, leaving less room for improvement. In spite of this, there were several significant findings that will help to inform future CDSS development.

Utilization of the CDSS for data entry, and thus the dynamic decision support functions, was low, although the utilization for reference information was higher. In addition, utilization was much higher among nurses and first year family medicine trainees. All of this suggests that, for this type of clinical scenario at least, the value of a CDSS is highest for its reference material and for its influence on developing new practice patterns and behaviors. Once those patterns and behaviors are developed, the CDSS is less appealing, likely in part as its recommendations do not change.

Our study aimed to improve our understanding of the impact of a CDSS on patient outcomes. Improvements in knowledge translation were observed from several perspectives. Self-reports of awareness and adoption of recommendations showed statistically significant improvement. Behaviors observed through chart audit showed that the assessment of pain specifically for neuropathic pain had a statistically significant increase as well. These outcomes are valuable as they are indicators of the quality of care. Ultimately, we would also hope to see an increase in the appropriate use of medication. We discovered that a majority of these patients were using first-line medications already at baseline, and that although there was a shift to even higher use of first-line medication in the year of observation, this was not statistically significant. As the utilization of the data fields in the CDSS was low, it seems unlikely that we can attribute any improved practice to the data collection aspect of the CDSS. Some combination of the training activities and the passive reference material included in the CDSS are more likely to have influenced practice.

We created a CDSS taking into account factors that have been shown to lead to the success of a CDSS, such as being created by study authors rather than by an external vendor, providing decision support at the time and location of decision making, and integration into practice workflow [2,13-15]. Our prototype included significant enhancements of the system interface that improved visibility and integration of the CDSS with usual work flow. This proved highly effective in usability testing [22]. However, production deployment of the modified interface was not complete by the time of this research project; therefore, optimal integration with the user interface was not achieved.

In addition, our CDSS is only minimally responsive to specific features of individual patients. Most features of this system would be typical of a recommendation system, rather than a decision support system. It may be most useful therefore as a training tool rather than for ongoing decision support. Neuropathic pain is a relatively rare entity in primary care, making it difficult to study [20]. Finally, our observational study design does not allow us to attribute any improved quality of care to the introduction of the CDSS itself.

Our study demonstrated that aligning all necessary dimensions of information systems development to meet research timelines, while achieving measurable impact on quality of care, is challenging. We were able to demonstrate improvement in clinical practice that may have resulted from clinicians developing practice patterns learned from recommendations included in the CDSS. Ongoing use of the CDSS was not common.