This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Background
“Lock to Live” (L2L) is a novel web-based decision aid for helping people at risk of suicide reduce access to firearms. Researchers have demonstrated that L2L is feasible to use and acceptable to patients, but little is known about how to implement L2L during web-based mental health care and in-person contact with clinicians.
Objective
The goal of this project was to support the implementation and evaluation of L2L during routine primary care and mental health specialty web-based and in-person encounters.
Methods
The L2L implementation and evaluation took place at Kaiser Permanente Washington (KPWA)—a large, regional, nonprofit health care system. Three dimensions from the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) model—Reach, Adoption, and Implementation—were selected to inform and evaluate the implementation of L2L at KPWA (January 1, 2020, to December 31, 2021). Electronic health record (EHR) data were used to purposefully recruit adult patients, including firearm owners and patients reporting suicidality, to participate in semistructured interviews. Interview themes were used to facilitate L2L implementation and inform subsequent semistructured interviews with clinicians responsible for suicide risk mitigation. Audio-recorded interviews were conducted via the web, transcribed, and coded, using a rapid qualitative inquiry approach. A descriptive analysis of EHR data was performed to summarize L2L reach and adoption among patients identified at high risk of suicide.
Results
The initial implementation consisted of updates for clinicians to add a URL and QR code referencing L2L to the safety planning EHR templates. Recommendations about introducing L2L were subsequently derived from the thematic analysis of semistructured interviews with patients (n=36), which included (1) “have an open conversation,” (2) “validate their situation,” (3) “share what to expect,” (4) “make it accessible and memorable,” and (5) “walk through the tool.” Clinicians’ interviews (n=30) showed a strong preference to have L2L included by default in the EHR-based safety planning template (in contrast to adding it manually). During the 2-year observation period, 2739 patients reported prior-month suicide attempt planning or intent and had a documented safety plan during the study period, including 745 (27.2%) who also received L2L. Over four 6-month subperiods of the observation period, L2L adoption rates increased substantially from 2% to 29% among primary care clinicians and from <1% to 48% among mental health clinicians.
Conclusions
Understanding the value of L2L from users’ perspectives was essential for facilitating implementation and increasing patient reach and clinician adoption. Incorporating L2L into the existing system-level, EHR-based safety plan template reduced the effort to use L2L and was likely the most impactful implementation strategy. As rising suicide rates galvanize the urgency of prevention, the findings from this project, including L2L implementation tools and strategies, will support efforts to promote safety for suicide prevention in health care nationwide.
suicide preventionfirearminternetimplementationsuicidepreventiondecision aidriskfeasiblesupportevaluationmental healthelectronic health recordtoolIntroduction
Firearm-related suicide accounts for approximately half of the suicide deaths in the United States annually [1]. Firearms are common in Americans’ lives [2]; about one-third of Americans report owning firearms [3], and an additional 10% report living in a household with a firearm [4], with higher rates in western states [2], among veterans [5], and in rural areas [6]. Moreover, the rate of ownership of new firearms appears to have increased recently among women, Black people, and Hispanic people [7]. Suicide attempts by firearm are highly lethal; researchers estimate that 85% to 95% of individuals who attempt suicide by firearm do not survive [8,9], and people with access to firearms, particularly if firearms are kept loaded and unlocked [10,11], have increased suicide risk [12,13]. Clinicians may have opportunities to intervene with patients at risk for firearm-related suicide because about 50% of individuals who die by suicide see a clinician in the month before death, and over 80% see one in the year before death [14]. Moreover, clinician-initiated discussions about reducing access to firearms have demonstrated effectiveness for improving firearm security practices (particularly in combination with free safe storage devices) [15-17], as well as promising findings for reducing suicide attempts [18,19].
Despite its potential benefits, clinician-initiated dialogue about limiting access to firearms is an uncommon practice across many primary care and mental health specialty practices [18,20]. Common barriers include time, clinicians’ unfamiliarity with firearms, and concerns about negatively impacting relationships or alienating patients [21]. “Lock to Live” (L2L) is a self-directed, anonymous, web-based decision aid that was designed to address these barriers. L2L was developed in collaboration with clinicians, firearm owners, and people who had experienced suicidal thoughts and attempts [22]. Consistent with international design standards [23,24], the L2L decision aid steps users through various considerations regarding in-home and out-of-home firearm storage options, such as types of storage, costs of storage, and background check requirements, with a goal of encouraging storage solution discussions that are consistent with the users’ values and preferences [22]. Two subsequent research studies demonstrated promising results for the feasibility and acceptability of offering L2L emergency care encounters [25] and for the uptake of L2L when it was offered via secure patient portal messages after outpatient care encounters [26]. Though L2L appears to be a useful tool for supporting suicide prevention in clinical practice, little is known about how to use L2L during routine health care encounters outside of a research context.
The goal of this project was to use mixed methods (qualitative and statistical evaluations) to support the implementation and evaluation of L2L during primary care and mental health specialty encounters in a large, regional health care system. Specifically, this project used semistructured interviews with clinicians and patients to support implementation, as well as statistical analyses to evaluate the reach and adoption of L2L over a 2-year period. The evaluation findings will inform considerations for implementing L2L nationwide to support suicide prevention in health care systems.
MethodsSetting
L2L implementation and evaluation took place at Kaiser Permanente Washington (KPWA)—1 of 8 regional Kaiser Permanente health care systems, which together form one of the nation’s largest nonprofit health care organizations and serve 12.5 million people [27]. At the time of this evaluation, KPWA had provided comprehensive medical care to approximately 700,000 members across Washington State via employer-sponsored insurance plans, individual insurance plans, or capitated Medicaid or Medicare programs. In 2016, KPWA augmented standard clinical workflows to support the identification and engagement of patients at high risk of suicide attempts (Figure 1) [28,29]. Specifically, a system-level electronic health record (EHR) template was created to support clinician-initiated safety planning among patients who are identified as at high risk of suicide during primary care and mental health specialty encounters [28,30]. Nationally, safety planning is a widely recommended best practice [31], and KPWA had an established process for safety planning that included addressing access to lethal means but did not offer any specific resources to clinicians or patients about firearm storage options. Consistent with the goal of L2L, step 6 of this safety plan template was designed to support patients in limiting access to lethal means, such as firearms and prescription medications.
Clinical workflow for supporting the identification and engagement of patients at high risk of suicide during primary and mental health specialty encounters at Kaiser Permanente Washington.
Implementation and Evaluation Framework, Data Sources, and Study Design
Three dimensions from the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) model—Reach, Adoption, and Implementation [32,33]—were selected to both inform and evaluate the implementation of L2L at KPWA (Multimedia Appendix 1) over a 2-year observation period (January 1, 2020, to December 31, 2021). Specifically, a qualitative, team-based, formative evaluation [34] was used, involving semistructured interviews with purposefully sampled patients and clinicians to facilitate implementation tools and strategies. Descriptive statistical analyses were used to evaluate the reach of L2L among patients identified at high risk of suicide and L2L adoption over a 2-year observation period. The findings were stratified by primary care and mental health specialty service settings due to the variation in the timing of the L2L trainings across these settings (Figure 2).
L2L implementation tools and strategies used over four 6-month subperiods of the 2-year observation period (January 1, 2020, to December 31, 2021). Tools are shown in blue, strategies are shown in green, and capped lines indicate semistructured interviews. EHR: electronic health record; KPWA: Kaiser Permanente Washington; L2L: Lock to Live; LICSW: licensed clinical social worker.
Semistructured Qualitative Interviews
EHR data were used to purposefully recruit adult patients to participate in semistructured interviews that included questions to elicit suggestions about introducing L2L, as part of a broader interview that focused on exploring perceptions regarding and experiences with firearm access assessment [35]. An invitation letter was mailed to sampled patients (age≥18 y) who had received a standardized question about firearm access (“Do you have access to guns? Yes/No”) in the prior 2 weeks on a mental health questionnaire [30]. A stratified sampling distribution was used to recruit approximately equal numbers of patients in 3 groups, including those who (1) reported firearm access, (2) reported no firearm access, and (3) did not respond (ie, left the question blank). The three sampling groups were also designed to purposefully include patients who had reported thoughts about self-harm in the prior 2 weeks via the ninth question of the 9-item Patient Health Questionnaire (PHQ-9) [36]. Interviewers attempted to reach all invitees for 2 weeks to invite them to participate in a phone interview. Following the portion of the interview guide that focused on firearm access assessment [35], interviewers described how patients reporting suicidal thoughts would soon be receiving L2L and elicited feedback about how to introduce this tool in a way that would make patients more likely to use it (Multimedia Appendix 2). This portion of the interview transcript was extracted into Excel (Microsoft Corporation) and analyzed by using a team-based, rapid, qualitative inquiry approach [37]. This involved an iterative data analysis wherein 2 coders independently coded the L2L portion of the transcript by using a combination of deductive and inductive content analyses, with codes developed a priori from the interview guide as well as codes that emerged from the interviews [38]. This was followed by several rounds of discussions with 2 additional team members who reviewed the coded data and reconciled themes iteratively for the purpose of using the summarized themes to facilitate implementation.
Following the completion of the qualitative analysis that focused on patient-informed L2L implementation, interviewers initiated clinician recruitment activities, which were also more broadly focused on firearm access assessment. At the recommendation of care delivery leaders, interviewers outreached to the following two groups of clinicians responsible for engaging patients identified at risk of suicide in risk mitigation (ie, safety planning [39]): (1) licensed clinical social workers (LICSWs) supporting integrated mental health in primary care [28] and (2) consulting nurses (registered nurses) responsible for connecting patients (ie, those reporting suicidality after business hours via telephone) to telephone-based follow-up care. The L2L portion of the interview guide included questions informed by patient interviews (Multimedia Appendix 2) and was analyzed by using the same rapid, qualitative inquiry approach [37] that was used for patient interviews for the purpose of further facilitating L2L implementation.
Descriptive Statistical AnalysesL2L Reach and Adoption
EHR data were used to summarize L2L “reach,” which was defined as the proportion of patients identified as at high risk of suicide via routine screening and assessment clinical workflows (Figure 1) and received the web-based decision aid. Specifically, we described characteristics of patients who had a documented safety plan during the 2-year observation period and characteristics of patients who had a safety plan that included a reference to L2L. Next, we described the adoption of L2L by primary care and mental health specialty clinicians over four 6-month subperiods of the observation period by calculating the proportions of patients identified at high risk of suicide (via suicide risk assessment; described in the Measures section) who had a documented safety plan with a reference to L2L and those who had a documented safety plan without a reference to L2L. We selected 6-month subperiods as the most helpful way to visually describe L2L adoption over time, since implementation paused during the initial COVID-19 outbreak (described in the Implementation Timeline, Tools, and Strategies section). We stratified by service setting due the variation in the timing of L2L trainings for these groups of clinicians.
Measures
The Columbia Suicide Severity Rating Scale (C-SSRS) was used to measure suicide risk, as per current clinical workflows. Specifically, patients reporting some level of suicide attempt planning or intent in the past month (ie, answering “yes” to C-SSRS question 3 or higher) were considered to be at “high risk” and alerted clinicians (via EHR prompts) to initiate safety planning. Distinctive phrases from standard EHR-based templates were used to detect safety plans documented in the text of clinical notes among patients identified at high risk (Multimedia Appendix 3). Sociodemographic and clinical characteristics of interest, including those known to be associated with firearm ownership and suicide risk [40,41], were measured by using the following administrative and diagnostic EHR data: age (continuous); sex (male or female); race and ethnicity (Asian, Black, Hispanic or Latinx, White, other, or unknown); insurance type (commercial, Medicare, Medicaid, or other); rurality (urban, large suburban, small suburban, or mostly rural) [42,43]; and prior year mental health, substance use, and self-harm diagnoses derived from the International Classification of Diseases, Tenth Revision, Clinical Modification. Reported firearm access was measured based on a positive response to the question on the mental health questionnaire [30] that was also used for qualitative interview recruitment (described in the Semistructured Qualitative Interviews section).
Ethical Considerations
The project team received approval from the KPWA Region Institutional Review Board (review number: 1826198) to conduct this study. Patients who agreed to participate in the phone interview provided oral consent, including permission for the interview to be audio-recorded and professionally transcribed, and they received a US $50 cash incentive for participation. During clinician recruitment activities, clinicians received up to 3 email invitations, which included a study information sheet and instructions for opting out of participation and further contact. Participating clinicians verbally consented to participation and received a US $50 gift card for participation.
ResultsImplementation Timeline, Tools, and Strategies
Over the 2-year implementation period (January 1, 2020, to December 31, 2021), a team of researchers and care delivery leaders took a pragmatic approach to iteratively creating and refining L2L implementation tools and strategies for primary care and mental health specialty clinicians (Figure 2). Initially, tools included an EHR-based macro (ie, EPIC SmartPhrase [Epic Systems Corporation]) for clinicians to easily add a URL and QR code referencing L2L to safety planning templates and a 1-page quick-start guide (ie, “Huddlecard”) with information on how to use the new SmartPhrase during routine clinic meetings (ie, “huddles”). In February 2020, the LICSWs who supported mental health care delivery in primary care [28] received information about L2L during a brief, web-based staff training session. Additional trainings that were planned for mental health specialty clinicians were put on hold during the widespread service disruption that subsequently occurred in response to the initial COVID-19 outbreak in March 2020. Additional tools and strategies were used, following recommendations from the care delivery leaders responsible for primary care and mental health service recovery and from the patients and clinicians who participated in semistructured qualitative interviews (detailed in the Findings From Semistructured Qualitative Interviews section).
Findings From Semistructured Qualitative Interviews
Of 76 patients who were purposefully sampled during 2 waves of recruitment, 36 were interviewed from November 18, 2019, to February 10, 2020 (Table 1). Five organizing themes were derived from the portion of the interview that elicited perceptions and suggestions about L2L and were used to create a handout for clinicians, with suggestions about how to introduce L2L to their patients at risk of suicide (Multimedia Appendix 4), including recommendations to (1) “have an open conversation,” (2) “validate their situation,” (3) “share what to expect,” (4) “make it accessible and memorable,” and (5) “walk through the tool” (Table 2). In addition to these recommendations, patients expressed a preference for receiving information about L2L from “trusted” and “caring” clinicians.
Characteristics of patient (n=36) and clinician (n=30) semistructured interview participants.
Patients
Clinicians
Sex, n (%)
Female
17 (47)
24 (80)
Male
19 (53)
6 (20)
Age (y), mean (SD)
47.3 (17.9)
44.3 (12.1)
Age category (y), n (%)
19-29
8 (22)
1 (3)
30-49
11 (31)
20 (67)
50-64
9 (25)
6 (20)
≥65
8 (22)
3 (10)
Race and ethnicity, n (%)
American Indian or Alaska Native
0 (0)
1 (3)
Black
3 (8)
2 (7)
Asian or Pacific Islander
3 (8)
5 (17)
Latinx or Hispanic
1 (3)
4 (13)
Unknown
2 (6)
0 (0)
White
27 (75)
18 (60)
Reported firearm accessa,b, n (%)
16 (44)
N/Ac
Reported thoughts about self-harm (prior 2 wk)a,d, n (%)
15 (42)
N/A
aPatients’ responses recorded on the Kaiser Permanente Washington mental health monitoring questionnaire used for criterion sampling within the 2 wk prior to the recruitment initiation.
b“Do you have access to guns? Yes/No.”
cN/A: not applicable.
dNinth question on the 9-item Patient Health Questionnaire: “Thoughts that you would be better off dead, or of hurting yourself.”
Thematic analysis of semistructured interviews with patients (n=36) and recommendations for introducing “Lock to Live” (L2L).
Themes
Illustrative quotesa
Recommendation
Show caring and compassion, ask permission, and respect autonomy
“I think it’s important to just take a breath, sit down with them, hold their hand, look them in the eye - ‘how can I help you? Help me help you. What’s going on? Tell me. What are you thinking? How are you feeling? How can I help you?’ Instead of an assembly line and ‘I only have a few minutes,’ so they [providers] don’t take the time.” (Patient B029)
“I would hope the provider is very warm and caring and explains it’s a safety precaution, it’s for your better health and it insures you’ll be safer…basically it’s another part of your little toolbox to keep yourself well.” (Patient A032)
“Probably compassionately, potentially generalized to begin with, to find out if the person is resistant right upfront… Maybe you could put a question, ‘would you be willing to consider options for storing or access to lethal means, whether it’s firearms or medication? Is it something you would be willing to discuss and look into if you were experiencing suicidal thoughts?’” (Patient A005)
“We’re offering you the means to protect yourself, this is not an us decision, this is a you decision. So here is the website, the online information and we encourage you to look at it, but it’s your decision….Nobody can force you to do things. So bringing it up more as like – not we’re taking it [firearm] away from you, but letting you decide what to do with it.” (Patient A024)
“Reassuring people that their firearm ownership will not end because they’re going through a rough patch in life; their ability to have their own authority to hold onto their possessions [will not end], firearms or not.” (Patient B036)
“Have an open conversation”: patients were more willing to listen and try a tool if a clinician took the time to connect, showed compassion for people’s unique experiences, and showed respect for autonomy.
Frame as helpful resource and normalize experiences
“I think overall education about the topic to start out with, just to say…‘this is what we have found is helpful, in these situations.’ Moving more into letting people know what their resources could be. Just education to be begin with, so it’s not so threatening. I think anytime somebody’s in a vulnerable place emotionally, they’re already possibly feeling threatened and they may not want to trust a lot of people.” (Patient A005)
“I would hope it would be pretty real, like a conversation…‘based on your health concerns you’re showing, we’ve got some important information we’d like to share with you,’ especially if that person has a relationship or feels responsible with the person presenting it, would stay there together and talk about it afterwards….Also statistics to help a person realize how more common this is.” (Patient A011)
“Validate their situation”: normalize their experience, share how common suicidal thoughts are, and be nonjudgmental in your approach; people have a variety of gun beliefs.
Address privacy and security
“[The provider] would have to explain what it does, how it’s going to work and how private it is - nobody can get into your part of the website anyway, your personal page, where you go. So she has to reassure them about that….I just don’t feel that being online is that secure.” (Patient B004)
“privacy is probably number one and an assurance that you’re not being turned in….I would be concerned in our surveillance state that disclosing things to a website about my firearm use might somehow come close to violating some kind of civil right to privacy.” (Patient B008)
“As long as people don’t have to put in information which can be tracked, I can see lots of people using it. I mean the minute [you have to enter] your name, address, phone number, medical ID number, whatever else, people are going to go – eh.” (Patient A033)
“Share what to expect”: address privacy and how information is stored if patients visit the website; assure patients that L2L is anonymous.
Accessibility is key
“You don’t want to make it hard to find on a website because it doesn’t take me very long. If something’s really hard to find on a website, I’m out of there.” (Patient B004)
“If there are hoops to jump through before you can access it, if you have to log in, go through a bunch of pages - maybe if it was right there, ready to access at any time, I’d say that’d be better.” (Patient B008)
“I’d love to see it everywhere. Have little cards that could be given out, a billboard, having my doctor [send it].” (Patient A033)
“If I knew it existed, I would probably try it. advertise it.” (Patient A032)
“Highlight it in your After Visit Summary too.” (Patient A002)
“Make it accessible and memorable”: have multiple routes for sharing the website and sending reminders (after-visit summary, message, website, pamphlet).
Demonstrate and “show, don’t just tell”
“I think when you’re in a pit of despair, to go and do it on your own, some people will do that and other people will not. They need to be taken by the hand and go, ‘what do you think about this?’ Read it together.” (Patient B036)
“I’m more keen to follow somebody who’s like ‘I’m offering you the opportunity to maybe do this together,’ instead of ‘I’m watching out for you.’” (Patient A024)
“Being shown an example would be nice, showing it off briefly. Knowing more specifically what it does or how it could be helpful as opposed to just knowing it exists.” (Patient B033)
“I think showing the patient or at least offering, would you like me to show you? Not just telling somebody, because short term memory is only like 30 s or a couple minutes and then you forget about it.” (Patient A002)
“Walk through the tool”: most patients said that a website walk-through, rather than simply having a conversation, would be helpful to overcome the barrier of trying something new, especially if already depressed.
aIdentifier “A”: patients in the first wave of interviews; identifier “B”: patients in the second wave of interviews (grammatic edits, noted in brackets, were added to clarify intended meaning).
Of 51 purposefully sampled clinicians responsible for safety planning with patients identified at high risk of suicide, 30 were interviewed from July 7, 2020, to October 8, 2020 (Table 1), including 25 LICSWs and 5 registered nurses. During the interviews with LICSWs, only 3 had actually used L2L with a patient—9 were unfamiliar with L2L, and 12 were familiar with but had not yet used L2L. Most clinicians saw clear benefits to L2L as an option for supporting both clinicians and patients. Several clinicians expressed concern about using the tool to replace dialogue about lethal means, and most supported the idea of a walk-through, as patients had recommended. Clinicians also expressed a strong preference to have L2L information included by default in the EHR-based safety planning template, in contrast to having clinicians remember to add it (via SmartPhrase). A clinician also suggested automatically including L2L in after-visit summaries when patients reported thoughts about self-harm on the PHQ-9. The implementation team worked with clinical partners to update the system-level, EHR-based safety plan template to include L2L information and updated the Huddlecard to communicate this change (Multimedia Appendix 5). After-visit summaries were used to provide safety plans to patients who were seen via a secure, web-based patient portal, and L2L was automatically included after the template change. Several clinicians also requested follow-up trainings or refreshers about L2L. The team therefore conducted a round of brief trainings, which were presented during routine clinic huddles with mental health specialty clinicians, and created a 3-minute training video (Multimedia Appendix 6).
Findings From Descriptive Statistical Analyses
During the study period, 2739 adult patients reported some prior-month suicide attempt planning or intent via routine suicide risk assessment workflows during primary care or mental health specialty encounters and had a documented safety plan, including 745 (27.2%) who also received L2L. Overall, there were no major differences in the demographic and clinical characteristics between patients who received L2L and the broader population that was identified as at risk of suicide and had a documented safety plan (Table 3).
Characteristics of patients who received “Lock to Live” (L2L; n=745) and were among patients with a documented safety plan (n=2739) during the implementation period (January 1, 2020, to December 31, 2021).
Patients who received L2L, n (%)
Patients with a documented safety plana, n (%)
Ageb (y)
18-39
513 (68.9)
1817 (66.3)
40-64
187 (25.1)
732 (26.7)
≥65
45 (6)
190 (6.9)
Sexc
Female
445 (59.7)
1753 (64)
Male
300 (40.3)
986 (36)
Race and ethnicityc
American Indian or Alaska Native
14 (1.9)
75 (2.7)
Asian
54 (7.2)
199 (7.3)
Black
52 (7)
166 (6.1)
Hawaiian or Pacific Islander
8 (1.1)
45 (1.6)
Hispanic or Latinx
59 (7.9)
201 (7.3)
Unknown
93 (12.5)
294 (10.7)
White
465 (62.4)
1759 (64.2)
Insurancec
Commercial
530 (71.1)
1831 (66.8)
Medicare
83 (11.1)
342 (12.5)
Medicaid
54 (7.2)
228 (8.3)
Not enrolled
78 (10.5)
338 (12.3)
Rural or urbanc,d
Urban
301 (40.4)
1010 (36.9)
Large suburban
205 (27.5)
799 (29.2)
Smaller suburban
186 (25)
802 (29.3)
Mostly rural
31 (4.2)
96 (3.5)
Mental health diagnosese
Depression
675 (90.6)
2502 (91.3)
Anxiety
652 (87.5)
2434 (88.9)
Serious mental illness
144 (19.3)
586 (21.4)
Substance use disorder
196 (26.3)
752 (27.5)
Suicide attempt diagnosis
39 (5.2)
175 (6.4)
Reported firearm accesse
150 (20.1)
501 (18.3)
aIncludes safety plans with L2L.
bAt evaluation midpoint (January 1, 2021).
cAt first encounter.
dMissing information for 22 patients.
eDuring implementation period.
The adoption of L2L increased substantially over the 2-year observation period (Tables 4 and 5). During this time, rates of documented safety plans among patients identified at high risk of suicide (C-SSRS score≥3) remained fairly consistent—51.2% to 55.2% of primary care patients and 73.4% to 78.4% of mental health specialty patients had a documented safety plan. However, over four 6-month subperiods of the observation period, L2L adoption rates increased substantially from 2% to 29% among primary care clinicians and <1% to 48% among mental health clinicians, increasing primarily after L2L was integrated into the EHR-based safety planning template.
Proportions of primary care patients who were identified as at high risk of suicide and had a documented safety plan during primary care encounters over the implementation period (January 1, 2020, to December 31, 2021).
Subperiods of implementation period
Patients with a documented safety plan that did not include “Lock to Live,” %
Patients with a documented safety plan that did include “Lock to Live,” %
Months 1-6
53.5
1.6
Months 7-12
50
4.1
Months 13-18
41.7
11.1
Months 19-24
22.2
29.1
Proportions of primary care patients who were identified as at high risk of suicide and had a documented safety plan during mental health specialty encounters over the implementation period (January 1, 2020, to December 31, 2021).
Subperiods of implementation period
Patients with a documented safety plan that did not include “Lock to Live,” %
Patients with a documented safety plan that did include “Lock to Live,” %
Months 1-6
78.1
0.3
Months 7-12
74.8
1.4
Months 13-18
53.8
19.6
Months 19-24
25.9
48.4
Discussion
This novel study used mixed methods to support the implementation and evaluation of a web-based decision aid that was designed to help patients at risk of suicide limit access to firearms. Specifically, findings from semistructured interviews with patients and clinicians were used to facilitate L2L implementation, while statistical analyses were used to describe rates of reach among patients identified at risk of suicide and increased adoption by clinicians who cared for them during the 2-year observation period.
L2L development centered users’ values and preferences in the design process [22]. Similarly, the tools and strategies developed for this project used information from semistructured interviews with people who were the most likely to be impacted by L2L implementation, including firearm owners, patients experiencing suicidality, and the clinicians who care for them. Clinicians have reported a lack of experience with handling firearms and have expressed apprehension about discussing firearm safety due to concerns about damaging relationships with patients [44-46]. Likewise, patients have expressed apprehension about disclosing access to firearms due to concerns about privacy, autonomy, and firearm ownership rights [47,48]. For these reasons, patients and clinicians perceive firearm access assessment as challenging but also as valuable for supporting suicide prevention [35]. This implementation project showed that clinicians, that is, those responsible for engaging at-risk primary care and mental health patients in suicide risk mitigation, willingly adopted the use of L2L to support safety planning.
This study also has important implementation implications. Unsurprisingly, the rates of L2L adoption increased after L2L was incorporated into the existing system-level safety planning template as a default (primarily in the latter half of year 2). This finding underscores the importance of removing barriers to the adoption of web-based decision aids and making adoption “easy” [49]. In contrast, those seeking change often focus on amplifying benefits or “selling” their new idea or innovation; however, it may be equally as important or more important to focus on “friction,” that is, “psychological forces that oppose and undermine change,” such as inertia, effort, emotion, and reactance [50]. In the case of L2L, reducing the effort required for clinicians to remember to use L2L appeared to be the main driver of its adoption. However, the tools and strategies that were designed to communicate about the benefits of using L2L (eg, training, video, Huddlecard, and newsletter information) were likely necessary for leaders to understand L2L’s value to patients and clinicians and approve the system-level change that was required to make L2L easier to use for clinicians.
This study has important clinical implications for supporting suicide prevention in health care. First, L2L supports clinicians who engage patients identified at risk of suicide in collaborative safety planning and lethal means counseling, which are evidence-based suicide risk mitigation practices that are recommended by the Zero Suicide Institute and follow the principles outlined in the National Strategy for Suicide Prevention [29,51-53]. Moreover, the recommendations from interview participants (“have an open conversation,” “share what to expect,” and “walk through the tool”) support a motivational interviewing approach to lethal means counseling and align with the recommendations of the Veterans Health Administration [54]. Second, L2L was developed by patients with lived experiences of suicidality and firearm ownership; therefore, L2L supports cultural competency in health care as a culturally aligned intervention [55]. Finally, this technology-based, EHR-embedded approach to addressing lethal means supports all 6 aims of health care quality that are outlined by the Institute of Medicine—safe, effective, patient-centered, timely, efficient, and equitable [56,57].
There are several limitations of this project that have implications for future research. First, the implementation of L2L at KPWA occurred during the initial outbreak of a global pandemic, which impacted the original implementation plans while health care systems responded to the pandemic and rapidly shifted toward providing web-based mental health care [58]. Semistructured interviews with patients took place prior to this shift. Future research should explore optimizing mental health care delivery workflows that support web-based suicide risk identification (ie, screening and assessment) [59] and incorporating L2L in web-based care encounters via secure telehealth platforms that are designed to support patient engagement. Second, L2L recognizes and addresses firearm policies related to background checks and how these policies might influence the legality of temporary firearm transfers for addressing suicide risk, but it does not address specific state laws. Additional work to understand the legality of recommendations about firearm safety practices may be helpful for health care systems that implement L2L. Third, this project was not designed to measure the specific impact of individual implementation strategies or determine whether L2L was effective in helping patients reduce access to firearms for suicide prevention purposes. Measuring the effectiveness of this tool, which was designed to support population-based suicide prevention, would require extending the implementation of L2L to other large health care systems nationwide and conducting other analyses that are designed to measure key functions of suicide prevention practices, including risk identification, engagement in evidence-based risk mitigation and treatment, and supportive care transitions [29]. Finally, L2L is meant to support adult patients at risk of suicide reduce access to firearms and other lethal means; additional tools and strategies are required to support youth at risk of suicide. Notably, there is a similar web-based decision aid that is available for this purpose; “Lock and Protect” was designed to help parents and caregivers reduce access to lethal means for youth suicide risk mitigation [60]. Similarly, the “Safety in Dementia” web-based decision aid was developed to support caregivers in addressing firearm access among individuals with Alzheimer disease and related dementias [61]. Future research should evaluate the implementation of these tools in routine care delivery.
In conclusion, incorporating L2L into the existing system-level safety plan template reduced the effort required to use L2L and was likely the most impactful implementation strategy for increasing clinician adoption and patient reach. However, understanding the value of L2L from the users’ perspectives was essential for effectively amplifying the suicide risk mitigation benefits. As rising suicide rates galvanize the urgency of prevention [62], the implementation tools and strategies developed for this project will be useful for health care systems nationwide.
This implementation evaluation was funded by Kaiser Permanente’s Office of Community Health and Center for Gun Violence Research and Education, as part of its Firearm Injury Prevention Program, and the Centers for Disease Control and Prevention (R01CE003460). The views and opinions expressed in this article are the responsibility of the authors and do not necessarily represent the official views of Kaiser Permanente or the Centers for Disease Control and Prevention. The authors acknowledge that this research would not be possible without the people who receive health care from Kaiser Permanente Washington and all the clinicians and staff who support the organization.
ReferencesWISQARS — your source for U.S. injury statisticsCenters for Disease Control and Prevention2022-04-26https://www.cdc.gov/injury/wisqars/facts.htmlMcCrackenHOkuleyHFloydLGun ownership in AmericaRAND Corporation2022-01-10https://www.rand.org/research/gun-policy/gun-ownership.htmlKalesanBVillarrealMDKeyesKMGaleaSGun ownership and social gun cultureInj Prev20160622321622010.1136/injuryprev-2015-04158626124073ParkerKHorowitzJMIgielnikROliphantJBBrownAAmerica’s complex relationship with gunsPew Research Center201706222022-03-29https://www.pewresearch.org/social-trends/2017/06/22/americas-complex-relationship-with-guns/ClevelandECAzraelDSimonettiJAMillerMFirearm ownership among American veterans: findings from the 2015 National Firearm SurveyInj Epidemiol201712194110.1186/s40621-017-0130-y29256160NordstromDLZwerlingCStromquistAMBurmeisterLFMerchantJARural population survey of behavioral and demographic risk factors for loaded firearmsInj Prev2001067211211610.1136/ip.7.2.11211428557MillerMZhangWAzraelDFirearm purchasing during the COVID-19 pandemic: results from the 2021 National Firearms SurveyAnn Intern Med202202175221922510.7326/M21-342334928699AnestisMDGuns and Suicide: An American Epidemic2018Oxford University PressCenters for Disease Control and Prevention, National Center for Injury Prevention and ControlFatal Injury Reports, National, Regional and State, 1981-2017, Web-Based Injury Statistics Query and Reporting System (WISQARS)2019Centers for Disease Control and PreventionKellermannALRivaraFPSomesGSuicide in the home in relation to gun ownershipN Engl J Med19920813327746747210.1056/NEJM1992081332707051308093ShenassaEDRogersMLSpaldingKLRobertsMBSafer storage of firearms at home and risk of suicide: a study of protective factors in a nationally representative sampleJ Epidemiol Community Health200410581084184810.1136/jech.2003.01734315365110AnglemyerAHorvathTRutherfordGThe accessibility of firearms and risk for suicide and homicide victimization among household members: a systematic review and meta-analysisAnn Intern Med20140121160210111010.7326/M13-130124592495MannJJMichelCAPrevention of firearm suicide in the United States: what works and what is possibleAm J Psychiatry20161011731096997910.1176/appi.ajp.2016.1601006927444796AhmedaniBKSimonGEStewartCHealth care contacts in the year before suicide deathJ Gen Intern Med20140629687087710.1007/s11606-014-2767-324567199Rowhani-RahbarASimonettiJARivaraFPEffectiveness of interventions to promote safe firearm storageEpidemiol Rev201638111112410.1093/epirev/mxv00626769724SimonettiJARowhani-RahbarAKingCBennettERivaraFPEvaluation of a community-based safe firearm and ammunition storage interventionInj Prev20180624321822310.1136/injuryprev-2016-04229228642248RunyanCWBeckerABrandspigelSBarberCTrudeauANovinsDLethal means counseling for parents of youth seeking emergency care for suicidalityWest J Emerg Med20160117181410.5811/westjem.2015.11.2859026823923BoggsJMBeckARitzwollerDPBattagliaCAndersonHDLindroothRCA quasi-experimental analysis of lethal means assessment and risk for subsequent suicide attempts and deathsJ Gen Intern Med2020063561709171410.1007/s11606-020-05641-432040838MonuteauxMCAzraelDMillerMAssociation of increased safe household firearm storage with firearm suicide and unintentional death among US youthsJAMA Pediatr2019071173765766210.1001/jamapediatrics.2019.107831081861BoggsJMQuintanaLMPowersJDHochbergSBeckAFrequency of clinicians’ assessments for access to lethal means in persons at risk for suicideArch Suicide Res202226112713610.1080/13811118.2020.176191732379012WaltersHKulkarniMFormanJRoederKTravisJValensteinMFeasibility and acceptability of interventions to delay gun access in VA mental health settingsGen Hosp Psychiatry201234669269810.1016/j.genhosppsych.2012.07.01222959420BetzMEKnoepkeCESiryB‘Lock to Live’: development of a firearm storage decision aid to enhance lethal means counselling and prevent suicideInj Prev20190925Suppl 1i18i2410.1136/injuryprev-2018-04294430317220StaceyDBennettCLBarryMJDecision aids for people facing health treatment or screening decisionsCochrane Database Syst Rev20111051010CD00143110.1002/14651858.CD001431.pub321975733LégaréFO’ConnorACGrahamISupporting patients facing difficult health care decisions: use of the Ottawa Decision Support FrameworkCan Fam Physician20060452447647717327891BetzMEKnoepkeCESimpsonSAn interactive web-based lethal means safety decision aid for suicidal adults (Lock to Live): pilot randomized controlled trialJ Med Internet Res20200129221e1625310.2196/1625332012056BoggsJMQuintanaLMBeckA"Lock to Live” for firearm and medication safety: feasibility and acceptability of a suicide prevention tool in a learning healthcare systemFront Digit Health2022096497415310.3389/fdgth.2022.97415336148209Fast factsKaiser Permanente2021-11-08https://about.kaiserpermanente.org/who-we-are/fast-factsRichardsJEParrishRLeeABradleyKCaldeiroRAn integrated care approach to identifying and treating the suicidal person in primary carePsychiatric Times2020-01-31https://www.psychiatrictimes.com/view/integrated-care-approach-identifying-and-treating-suicidal-person-primary-careRichardsJESimonGEBoggsJMAn implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicideImplement Res Pract202101122633489521101176910.1177/2633489521101176934447940RichardsJEKuoEStewartCSelf-reported access to firearms among patients receiving care for mental health and substance useJAMA Health Forum202108628e21197310.1001/jamahealthforum.2021.197335977197National Academies of Sciences, Engineering, and MedicineHealth Systems Interventions to Prevent Firearm Injuries and Death: Proceedings of a Workshop2019The National Academies Press10.17226/25354GlasgowREVogtTMBolesSMEvaluating the public health impact of health promotion interventions: the RE-AIM frameworkAm J Public Health1999098991322132710.2105/ajph.89.9.132210474547HoltropJSRabinBAGlasgowREQualitative approaches to use of the RE-AIM framework: rationale and methodsBMC Health Serv Res2018031318117710.1186/s12913-018-2938-829534729StetlerCBLegroMWWallaceCMThe role of formative evaluation in implementation research and the QUERI experienceJ Gen Intern Med20060221Suppl 2S1S810.1111/j.1525-1497.2006.00355.x16637954RichardsJEKuoESWhitesideUPatient and clinician perspectives of a standardized question about firearm access to support suicide prevention: a qualitative studyJAMA Health Forum2022114311e22425210.1001/jamahealthforum.2022.425236416815KroenkeKSpitzerRLWilliamsJBThe PHQ-9: validity of a brief depression severity measureJ Gen Intern Med20010916960661310.1046/j.1525-1497.2001.016009606.x11556941BeebeJRapid Qualitative Inquiry: A Field Guide to Team-Based Assessment2014Rowman & LittlefieldHsiehHFShannonSEThree approaches to qualitative content analysisQual Health Res2005111591277128810.1177/104973230527668716204405StanleyBBrownGKSafety planning intervention: a brief intervention to mitigate suicide riskCogn Behav Pract20120519225626410.1016/j.cbpra.2011.01.001NockMKBorgesGBrometEJChaCBKesslerRCLeeSSuicide and suicidal behaviorEpidemiol Rev200830113315410.1093/epirev/mxn00218653727MorganERGomezARowhani-RahbarAFirearm ownership, storage practices, and suicide risk factors in Washington State, 2013-2016Am J Public Health201807108788288810.2105/AJPH.2018.30440329771611GoldstickJECarterPMCunninghamRMCurrent epidemiological trends in firearm mortality in the United StatesJAMA Psychiatry202103178324124210.1001/jamapsychiatry.2020.298632965479IngramDDFrancoSJ2013 NCHS Urban-Rural Classification Scheme for CountiesVital Health Stat 220140416617324776070KettererARPolandSRayKAbuhasiraRAldeenAZEmergency providers’ familiarity with firearms: a national surveyAcad Emerg Med20200327318519410.1111/acem.1384931957230FarcyDADoriaNMoreno-WaltonLEmergency physician survey on firearm injury prevention: where can we improve?West J Emerg Med202102822225726510.5811/westjem.2020.11.4928333856309WintemuteGJBetzMERanneyMLYou can: physicians, patients, and firearmsAnn Intern Med2016082165320521310.7326/M15-290527183181RichardsJEHohlSDSegalCD“What will happen if I say yes?” perspectives on a standardized firearm access question among adults with depressive symptomsPsychiatr Serv202108172889890410.1176/appi.ps.20200018733940947KhazanovGKKeddemSHoskinsKStakeholder perceptions of lethal means safety counseling: a qualitative systematic reviewFront Psychiatry202210201399341510.3389/fpsyt.2022.99341536339871ServiceOHallsworthMHalpernDEAST: four simple ways to apply behavioural insightsThe Behavioural Insights Team2023-10-12https://www.bi.team/wp-content/uploads/2015/07/BIT-Publication-EAST_FA_WEB.pdfNordgrenLSchonthalDThe Human Element: Overcoming the Resistance That Awaits New Ideas2021WileyZero Suicide InstituteEducation Development Center2023-10-12https://solutions.edc.org/solutions/zero-suicide-instituteU.S. Department of Health and Human Services (HHS) Office of the Surgeon General and National Action Alliance for Suicide Prevention2012 National Strategy for Suicide Prevention: Goals and Objectives for Action2012HHSLaymanDMKammerJLeckman-WestinEThe relationship between suicidal behaviors and Zero Suicide organizational best practices in outpatient mental health clinicsPsychiatr Serv202110172101118112510.1176/appi.ps.20200052533730886Lethal means safety counselingUS Department of Veterans Affairs2023-10-12https://www.mirecc.va.gov/visn19/lethalmeanssafety/counseling/TIP 59: improving cultural competenceSubstance Abuse and Mental Health Services Administration20152023-10-12https://store.samhsa.gov/product/TIP-59-Improving-Cultural-Competence/SMA15-4849Institute of Medicine (US) Committee on Quality of Health Care in AmericaCrossing the Quality Chasm: A New Health System for the 21st Century2001National Academies Press10.17226/10027Six domains of healthcare qualityAgency for Healthcare Research and Quality2015022024-03-20https://www.ahrq.gov/talkingquality/measures/six-domains.htmlWosikJFudimMCameronBTelehealth transformation: COVID-19 and the rise of virtual careJ Am Med Inform Assoc202006127695796210.1093/jamia/ocaa06732311034SimonGEStewartCCGaryMCRichardsJEDetecting and assessing suicide ideation during the COVID-19 pandemicJt Comm J Qual Patient Saf20210747745245710.1016/j.jcjq.2021.04.00233994334AsarnowJRZulloLErnestusSM“Lock and Protect”: development of a digital decision aid to support lethal means counseling in parents of suicidal youthFront Psychiatry20211061273623610.3389/fpsyt.2021.73623634690841McCarthyVPortzJFischerSMA web-based decision aid for caregivers of persons with dementia with firearm access (Safe at Home study): protocol for a randomized controlled trialJMIR Res Protoc2023013112e4370210.2196/4370236719721BarryEFollowing a two-year decline, suicide rates rose again in 2021The New York Times202302112023-02-13https://www.nytimes.com/2023/02/11/health/suicide-rates-cdc.html#:~:text=Suicide%20increased%20among%20younger%20Black,reported
The three dimensions of RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) selected to inform and evaluate the implementation of “Lock to Live.”
Patient and clinician interview questions that focused on “Lock to Live” implementation.
Elements included in the search for safety plans documented in clinical notes text.
Introducing Lock2Live.org: a guide for clinicians.
“Lock to Live” Huddlecard.
“Lock to Live” video for Kaiser Permanente Washington clinicians.