This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.
Electronic referrals or e-referrals can be defined as the electronic transmission of patient data and clinical requests between health service providers. National electronic referral systems have proved challenging to implement due to problems of fit between the technical systems proposed and the existing sociotechnical systems. In seeming contradiction to a sociotechnical approach, the Irish Health Service Executive initiated an incremental implementation of a National Electronic Referral Programme (NERP), with step 1 including only the technical capability for general practitioners to submit electronic referral requests to hospital outpatient departments. The technology component of the program was specified, but any changes required to embed that technology in the existing sociotechnical system were not specified.
This study aimed to theoretically frame the lessons learned from the NERP step 1 on the design and implementation of a national health information technology program.
A case study design was employed, using qualitative interviews with key stakeholders of the NERP step 1 (N=41). A theory-driven thematic analysis of the interview data was conducted, using Barker et al’s
The NERP step 1 was broadly welcomed by key stakeholders as the first step in the implementation of electronic referrals—delivering improvements in the speed, completeness of demographic information, and legibility and traceability of referral requests. National leadership and digitalized health records in general practice were critical enabling factors. Inhibiting factors included policy uncertainty about the future organizational structures within which electronic referrals would be implemented; the need to establish a central referral office consistent with these organizational structures; outstanding interoperability issues between the electronic referral solution and hospital patient administration systems; and an anticipated need to develop specialist referral templates for some specialties. A lack of specification of the sociotechnical elements of the NERP step 1 inhibited the necessary testing and refinement of the change package used to implement the program.
The key strengths of the NERP step 1 are patient safety benefits. The NERP was progressed beyond the pilot stage despite limited resources and outstanding interoperability issues. In addition, a new electronic health unit in Ireland (
Electronic referrals or e-referrals can be defined as the electronic transmission of patient data and clinical requests between health service providers [
The initiation of an Irish electronic referral pilot program in January 2011 was not solely motivated by the potential for electronic referrals to transform the interface between primary and specialty care. A crisis emerged in March 2010, when the media reported that one of Ireland’s largest hospitals had 30,000 unopened or unprocessed GP outpatient referrals. The Irish Health Service Executive (HSE) commissioned an investigation [
Meanwhile, an advisory group had been established in the South of the country, comprising clinical, management, information technology (IT), and patient representatives, to reconfigure hospital services in that region. The group partnered with
Despite these obstacles to the implementation of an end-to-end electronic referral solution, the pilot project successfully established the technical capability, through the
These simple and yet important patient safety benefits informed a decision by a newly established unit in the HSE, called
Reviews from some of the earliest deployments of national electronic referral systems, including Norway [
In the UK, Eason applied a sociotechnical lens to the implementation of the National Health Service’s National Programme for IT and highlighted that in national health IT programs, the desired technical and social systems are not designed and implemented simultaneously [
Theoretical framework—Barker et al’s [
Instead, standard technical systems are predefined at the national level, and flexibility needs to be provided for local implementation sites to adopt technical systems in ways that meet local needs and enable them to engage in sociotechnical systems design at a level where the local user community can play a full part [
To contribute to this discussion on the design and implementation of national health IT programs, this paper presents the findings from qualitative, in-depth interviews conducted with key stakeholders in the implementation of the NERP step 1. The following two research questions seek to explore the arguments for and against progressing with the scale-up of only the technical elements of a national health IT program, using Ireland’s NERP step 1 as an empirical case study: (1) What were the strengths and limitations of the scale-up of the NERP step 1, as a technical-only intervention? and (2) Do the sociotechnical elements of a national health IT program need to be specified at the national level?
This study aims to theoretically frame the lessons learned from Ireland’s NERP step 1 for policy makers and implementers seeking guidance on how to design and implement national health IT programs.
We adopted the Institute for Healthcare Improvement’s
The
An important reason for selecting this theoretical framework is that it can accommodate the NERP’s incremental design, whereby this study only examines step 1 in the implementation of electronic referrals and not the complete implementation of electronic referrals. The
This study explored the implementation of the NERP step 1 using qualitative, in-depth interviews with key program stakeholders. This approach captures individual participants’ experiences, narratives, ideas, and discourses [
Ethical approval was granted by the Office of Research Ethics in University College Dublin (UCD). No vulnerable populations participated in this study, and no patient data were collected. All participants were interviewed in a professional capacity as stakeholders in the scale-up of electronic referrals in Ireland. Participant anonymity and confidential data management were the dominant ethical considerations for this study and were maintained in line with UCD Research Ethics Guidelines.
This study investigated the lessons to be learned from the NERP step 1 on scaling-up only the technical elements of a national health IT program. Access to 1 of 7 pilot sites and 5 of 42 sites targeted by the NERP step 1 was facilitated by
The following participants were recruited from the pilot site: pilot management (n=3); hospital administration or management (n=3); general practice (n=3); and information communication technology (ICT; n=3). In addition, the pilot general practice and ICT stakeholders were involved in the NERP step 1 and therefore appear in
Overall, 28 interviews were scheduled. Of 41 participants, 19 participated in face-to-face group interviews and the remaining 22 were interviewed individually. The group interviews involved a range of 2-5 participants and were predominantly undertaken with the hospital administration or management stakeholder group. Of 22 interviews conducted with individual participants, 5 were conducted via telephone and 17 on a face-to-face basis. All participants consented to have their interview recorded at the outset, using a digital voice recorder.
We conducted a thematic analysis, using Braun and Clarke’s [
Participant involvement in the National Electronic Referral Programme (NERP; N=41).
Key stakeholder types | Involvement, n (%) | |||
Pilot (n=12) | NERP (n=35) | Both (n=6) | ||
Pilot management | 3 (25) | — | — | |
Hospital administration or management | 3 (25) | — | — | |
NERP management | — | 2 (6) | — | |
Health Service Executive | — | 3 (8) | — | |
Information communication technology | 3 (25) | 3 (8) | 3 (50) | |
General practice | 3 (25) | 3 (8) | 3 (50) | |
Hospital administration or management | — | 17 (49) | — | |
Information communication technology | — | 3 (8) | — | |
General practice | — | 4 (11) | — |
Furthermore, any points of divergence in the interpretation of how the spoken word should be written in the transcripts were documented and later discussed by MMG and GM to obtain an agreement on transcription.
In the first analytical step, we conducted an inductive thematic analysis [
In the second analytical step, Barker et al’s
This section presents data collected via qualitative in-depth interviews with key stakeholders of the NERP step 1 using the theoretical structure provided by Barker et al’s
Finally, to provide some context for these results,
The
...an easy sell, [its] patient safety...a solution has been developed so it’s a matter of taking the solution and rolling it out to different acute hospitals.
Similar to the pilot experience, the NERP step 1 participants cited speed, complete demographic information, legibility, and traceability as the 4 key patient safety improvements delivered in the NERP step 1. Participants commented that
It’s instant, it sells itself. You send in the referral, the hospital has it, there’s no post, you’re not waiting a day for it to be delivered.
It’s good they have a minimum data set...Like we’ll never be missing a date of birth or [receive]...only one line of an address...They will always give you a phone number on it.
Another hospital administrator pointed out that full contact details, including mobile phone numbers, are very important so that they can
It’s legible you know. Many times you have to ring them up [GP surgeries].
However, with electronic referrals, they
One important sociotechnical element built into the pilot program had been the requirement for hospitals to return a triage outcome message, via
...it remains to be seen...how negative that will be...It reduces the communication back to the GP, and it doesn’t tell the GP how the patient has been triaged.
A national-level participant commented that:
...we thought the GPs would be up in arms and they would go crazy about it but actually when we did go back to the ICGP...they said, “we’d be disappointed but at the same time...we would prefer that they [hospitals] went with it without responses than not go at all.”
The stakeholder map. GP: general practitioner, ICT: information communication technology, NERP: National Electronic Referral Programme.
Application of the
As such, GPs supported the NERP step 1 in proceeding with the scale-up of the technical capability for all hospitals to receive electronic referral requests, with the knowledge that hospitals did not have the “human capability for scale-up” (
Set-Up is the first
When we are asked to rollout eReferrals within 12 months, what we can do is we can put the capability in place for each of the sites.
In terms of the “ask” to rollout electronic referrals in 12 months, the “leadership” provided by the HSE’s new
If he hadn’t arrived I would say that at this stage, we’d have rolled it out in the South or Southwest Hospital Group [pilot] and possibly no further.
A national-level participant commented that:
...this is the first...of any of the projects we’ve done, where there’s been a national focus. Where from the top, it’s been said, “everyone has to accept electronic referrals by X date.”
In addition, a GP who was involved in the pilot commented that the new CIO was
...there’s a huge amount of work that needs to be done and huge investment that needs to take place, and I suppose that remains to be seen, whether that will be available.
This last comment suggested that leadership requires not only vision but also the ability to secure funding. A comment by a national-level stakeholder supported this suggestion in referring to the
...if we can deliver a project of this type in a timeline that’s considered sensible...then it provides more confidence that the Office of the CIO and the Healthlink team can actually deliver significant change in a reasonable amount of time.
This comment provided context for the ambitious 12-month time-frame to scale-up the technical capability for electronic referrals, particularly because further resources need to be secured to proceed beyond step 1 of the NERP.
Second, defining the ambition of the NERP step 1 as putting the technical “capability” in place for all hospitals to receive electronic referral requests from GPs indicated confidence that GPs will submit their referrals electronically, should this facility be available to them. The level of digitalization of general practice health records represents an important “infrastructure for scale-up”
[GPs] have all the information in the system and being able to extract it and package it up and send it off electronically is kind of a side effect of the investment that they’ve [GPs] made over the years.
No formal incentives are offered to GPs to use electronic referral requests and, therefore, their adoption of the solution relies upon their technical capability to submit electronic referrals and the perception that this solution is a “better idea.”
The second
Implementing this first element was the focus of the NERP step 1 throughout this study period (October 2015-May 2016) and was achieved in May 2016, after 17 months of implementation, when (1) all public hospitals had (2) at least one specialty accepting (3) outpatient electronic referrals from (4) referring GPs. With all hospitals targeted, a minimal specification of at least one OPD specialty in each hospital can be assumed. However, no specification was provided for the target proportion of electronic versus paper referral requests or the proportion of GPs targeted to use electronic referral requests. The remainder of the description of the NERP step 1’s progression through the
Interview data collected suggested that the incomplete development of the scalable unit reflects uncertainty in national “Policy” (
[It is] not something that IT can make a call on...we’ll certainly drive it once we’re clear this is a direction that is best for the patient and for the service.
This quote illustrated a governance challenge faced by national health IT programs like NERP, in that the authority to make key decisions about the design of such programs might lie outside of the program team.
In addition, shortcomings emerged at this phase in the “communication”
...there are meetings and different groups but...the administrative end is not heard all the time...they come looking for secretarial support but then no budget or nothing available.
Perhaps, stemming from this lack of consultation, there was an unmet expectation from CRO staff that electronic referrals were going to save them time.
Everything that’s been done electronically, it’s supposed to save time and resources but actually it doesn’t. It does exactly the opposite mostly.
While electronic referrals will reduce the administrative burden for CRO staff once the interoperability issues between
Furthermore, participants reported that the strategy for communicating the value of electronic referrals to GPs requires clarification to increase the proportion of GPs submitting electronic referral requests. This element of the scalable unit was not specified for the NERP step 1, and therefore, nobody was officially tasked with the responsibility to increase the proportion of GPs using electronic referrals. A hospital-based implementer suggested that:
I felt like why was I having to try to promote Healthlink?...Nobody could give me any communication tools to use for the GPs—so we had to try and figure out the best way to do it.
National-level participants suggested that ultimately, GPs are
That is really, a HSE hospital led kind of initiative...[The] ultimate step would be for management to say, “this is how we want you to refer”...unless there is an exception.
In this study, participant GPs broadly agreed with this perspective, suggesting that local hospitals are in the best position to change GPs’ referral behaviors together with local peer promotion through the ICGP’s Continuing Medical Education meetings. GPs commented that:
I think it would be great to see the hospitals running with the ball on this one alright.
They also said,
Test Scale-Up is where the underlying theory of change and the change package are tested in a broader range of settings to refine program hypotheses and build the belief and will of leaders and frontline staff to support the changes [
...giving greater visibility on referral volume, referral tracking, all those sorts of things by specialty within hospitals.
...whereas before we were relying on staff members putting them in an Excel...So now every referral to be processed must be on PAS.
The disadvantage of structured messaging is that it might limit GPs’ ability to communicate details about a referral. One GP commented that:
You can write a very good clinical note using free text, probably the best quality clinical notes because it captures what the patient and yourself are saying. You can’t do that with something that’s completely structured. When you’re picking from drop-down menus or whatever.
Moreover, participants highlighted the importance of buy-in from stakeholders on the type of data collected. One participant commented that:
I maintain that no clinician wants to work to a political target...They don’t mean anything clinically
A pilot participant cautioned that:
...when you use data in a punitive way...people are resistant to it.
These quotes highlighted the potential for electronic referrals to greatly improve the volume and quality of data collected on referral management as well as the importance of engaging with stakeholders to determine what data would offer the most constructive and meaningful insights for the quality improvement.
Regarding reporting systems, participants reported receiving a monthly
It has actually highlighted that we weren’t doing it [managing referrals] as well as we thought we were doing it.
Participants described two other national programs to which they submit data and receive reports relevant to electronic referrals, namely the HSE’s Outpatient Services Performance Improvement Programme (OSPIP) and an independent statutory body called the National Treatment Purchase Fund (NTPF). Crucially, however, neither OSPIP nor NTPF targets are formally aligned with any specific targets for the NERP step 1. A CRO participant commented that “in the Health Service, there’s no picture of what’s happening” (Participant 22).
This suggests a lack of data feedback to implementers, either on the NERP step 1 on its own, or a more strategic data reporting system that utilizes the data collected across HSE and statutory programs.
This lack of development in data collection and reporting systems exerts knock-on effects on the “learning system” (
...normally suggest[ed] that they [implementers] speak to other counterparts in other hospitals that have already gone live.
Furthermore, one implementation site participant commented that he had
Besides these support systems, an important
...some of them are not open to new stuff because they’ve been burnt in the past...Most sites need reassurance as to the impact it’s [NERP step 1] likely to have operationally for them.
Conversely, CRO participants reported that while they were cautious about electronic referrals, now that they are using
Now that we know how easy it is to go electronic, it would be amazing to cut out all the filling.
Similarly, another CRO participant commented that:
...rather than sitting on this for a year and everyone would just get too complacent with it and then it’s more change...If you’re in the middle of a project and there’s more coming on board, you just take it.
These comments highlighted the importance of developing a complete, scalable unit, whereby participants are clear on what the vision for full scale is and they can then maintain momentum in going to full scale.
Go to Full Scale is the fourth and final phase of the
Policy uncertainty is a key sustainability issue for the NERP step 1, which has already been described above as the uncertainty about whether to proceed with the sociotechnical process changes for electronic referrals at the hospital level [
...resources are still an issue with the Central Booking and the Central Office. So, to do it from within your current compliment [of staff] initially is difficult.
Similarly, another CRO participant commented that:
We do [have a CRO] only we have no one to sit in it. That’s why it comes to me. I’m the central office.
The variation in terminology used by participants to refer to the CRO in the above quotes reflects the variation in set-up and functions of these offices across sites. This variation helps to explain why some hospitals experience greater difficulty than others in implementing electronic referrals, if their administrative staff has not been reconfigured into a CRO.
The third key sustainability issue for the NERP is the persisting interoperability issues between
...it is a great system (new iPMS)...if the correct processes were in place, it would be perfect.
The process changes involved in implementing iPMS require CRO staff training. CRO staff reported that while the HSE IPMS team did train on-site trainers, these
Furthermore, consultant engagement was beyond the scope of the NERP step 1 because electronic referrals were printed once they reached the hospital. Implementing a more complete electronic referral solution will require hospital consultants triaging electronic referral requests online. An implementation site participant explained that they have had consultants from certain specialties requesting that
...well Healthlink said they would take on certain forms but if we could just run with this...and see how we get on with it.
Similarly, a national-level participant commented that:
...we get a consistent message from the hospitals that they’d like to do more in the way of specialist referral.
The challenge is that consultants throughout the country
I know that Healthlink did have some issues with [specialist referral forms]. They just want to consolidate as much as possible.
The
Engaging with the vendors is a challenge...because we’re very reliant on them doing the initial work to get their products modified.
This study aimed to theoretically frame the lessons learned from the NERP step 1 on the design and implementation of a national health IT program. The NERP step 1 presented an interesting case study of implementing a national health IT program because it explicitly committed to a technical-first implementation rather than a sociotechnical approach. A key strength of the program was that it was welcomed by most key stakeholders as the first step in the implementation of electronic referrals, delivering important patient safety benefits. A national implementation of electronic referrals was progressed, despite limited resources and outstanding interoperability issues. In addition, it gained credibility for a new
The first research question posed by this paper asked what are the strengths and limitations of the scale-up of the NERP step 1 as a technical-only intervention. A key strength of the NERP step 1 is that it scaled-up the technical capability of GPs to submit electronic referral requests to at least one OPD specialty in all public hospitals. The four patient safety improvements reported by the NERP step 1 participants, including speed of transfer, more complete demographic information, legibility, and traceability, have been recognized internationally as key benefits of implementing an electronic referral solution [
Associated with this institutional complexity, a key limitation of the NERP step 1 was that it was poorly integrated within the wider policy and quality improvement agenda of the health service. The program was designed and implemented by
The second, related limitation of the NERP step 1 was the incomplete specification of the program’s scalable unit. Once all hospitals had at least one OPD specialty accepting electronic referral requests from GPs, the single objective for step 1 of the program was achieved. In this study, participants reported an implementer burden owing to this lack of specification. Hospital participants reported having to try and figure out for themselves how to engage local GPs, although they did not consider this their responsibility. Some hospital administrators expressed dissatisfaction with the low level of consultation, which inhibited them from communicating the “double-jobbing” challenges associated with sending the GP triage outcome message. Sending this message was a feature of the pilot project but not the NERP step 1; however, interviews with hospital administrators suggested that the scope of the NERP step 1 was neither clearly specified nor communicated to them. This limited program specification also restricted the potential to develop data collection and reporting systems, through which individual implementation sites could monitor their progress [
The second research question asked whether the sociotechnical elements of a large-scale national health IT program need to be specified at the national policy level. Analyzing the NERP step 1 using the
The wider literature is highly critical of designing national health IT programs with rigid top-down change packages that do not leave space for local adaption. Coiera, for instance, argued that centrally defined, top-down implementations of national health IT programs become increasingly out of step with service needs, and clinical providers will have to build workarounds to make the aging system meet emerging needs [
As such, the academic literature’s advocacy of a sociotechnical approach to implementing national health IT programs is not contradicted by this study. This paper started with an observation that the NERP step 1 was initiated with a technical-only intervention and uncertainty about whether this type of implementation strategy [
Barker et al’s
A key limitation of this study is that hospital specialists were not interviewed. Hospital specialists were not formally engaged in the design or implementation of the NERP step 1 and therefore were not considered key informants in this early stage of the program. Upon the receipt of an electronic referral request, the Hospital CRO prints the electronic referral request. By the time it reaches a specialist, it is a paper-based referral request, just like any other. The only change encountered by specialists is that electronic referral requests from GPs are presented to them for triage on a standardized template. The triage phase of referral management was not included in the NERP step 1, and therefore, specialist dissatisfaction or satisfaction with the standardized GP-OPD referral template was beyond the remit of this study. Issues were raised within the study, however, for which it would have been valuable to have obtained a specialist medical perspective. These include the centralization of the referral management to a CRO at a hospital or hospital group level or the suitability of using the standardized GP-OPD referral template for all OPD specialties. Hence, future research should focus on these issues as they relate to the later stages of the
The second limitation is that the participants were not recruited from randomly selected implementation sites. Access to 1 pilot site and 5 NERP step 1 implementation sites was arranged through
This qualitative study of the early-stage implementation of the NERP provides empirical insights into the complexity of implementing a national health IT program. The incremental design of this program—with step 1 only seeking to scale-up the technical capability for the e-request phase of an electronic referral solution—made the NERP step 1 an interesting case study from a sociotechnical perspective.
The strengths of this implementation were that it successfully scaled-up the technical capability for GPs to submit electronic referral requests to at least one specialty in all hospitals in the Irish public health system. In addition, it maintained progress in the implementation of an electronic referral solution beyond piloting despite limited resources and outstanding interoperability issues. Finally, it built credibility and confidence in the new
These limitations were a consequence of not specifying a complete scalable unit, including the sociotechnical elements of the program. In conclusion, although the sociotechnical elements of a program do not have to be specified in tandem with technical elements, they do need to be specified quite early in the implementation process so that the potential change packages for implementing the scalable unit can be tested and refined into a scalable set of interventions.
Number of electronic referral requests submitted to participating hospital outpatient departments in 2015/2016. Data collection period is Oct 2015-May 2016. Source:
Monthly number of electronic referral requests (Sep 2015-Feb 2018). Source:
Upward trend in % of general electronic referrals to outpatient departments (Mar 2016-Feb 2018). Source:
chief information officer
central referral office
general practitioner
General Practice Information Technology Group (Ireland)
Health Information and Quality Authority (Ireland)
Health Service Executive (Ireland)
Irish College of General Practitioners (Ireland)
information communication technology
Integrated Patient Management System
Integrated Patient Management System Programme (Ireland)
information technology
National Treatment Purchase Fund (Ireland)
National Electronic Referral Programme (Ireland)
outpatient department
Outpatient Services Performance Improvement Programme (Ireland)
patient administrative system
Plan-Do-Study-Act cycle of quality improvement
University College Dublin
The authors wish to sincerely thank the 41 key stakeholders who kindly gave their time to participate in this study. We also thank
None declared.