This paper is in the following e-collection/theme issue:

Information Seeking, Information Needs (465) Public Health (197) eHealth Infrastructures (345) Depression and Mood Disorders; Suicide Prevention (1363) Intentional Self-Harm (96) Web-based and Mobile Health Interventions (3297)

Published on in Vol 12 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/48007, first published .
Reducing Firearm Access for Suicide Prevention: Implementation Evaluation of the Web-Based “Lock to Live” Decision Aid in Routine Health Care Encounters

Reducing Firearm Access for Suicide Prevention: Implementation Evaluation of the Web-Based “Lock to Live” Decision Aid in Routine Health Care Encounters

Reducing Firearm Access for Suicide Prevention: Implementation Evaluation of the Web-Based “Lock to Live” Decision Aid in Routine Health Care Encounters

Authors of this article:

Julie Angerhofer Richards1, 2 Author Orcid Image ;   Elena Kuo1 Author Orcid Image ;   Christine Stewart1 Author Orcid Image ;   Lisa Shulman1 Author Orcid Image ;   Rebecca Parrish3 Author Orcid Image ;   Ursula Whiteside4, 5 Author Orcid Image ;   Jennifer M Boggs6 Author Orcid Image ;   Gregory E Simon1, 3, 5 Author Orcid Image ;   Ali Rowhani-Rahbar7, 8 Author Orcid Image ;   Marian E Betz9 Author Orcid Image
Article Authors Cited by Tweetations (13) Metrics

    1Kaiser Permanente Washington Health Research Institute, , Seattle, WA, , United States

    2Department of Health Systems and Population Health, University of Washington, , Seattle, WA, , United States

    3Department of Mental Health & Wellness, Kaiser Permanente Washington, , Seattle, WA, , United States

    4NowMattersNow.org, , Seattle, WA, , United States

    5Department of Psychiatry and Behavioral Sciences, University of Washington, , Seattle, WA, , United States

    6Kaiser Permanente Colorado Institute for Health Research, , Aurora, CO, , United States

    7Department of Epidemiology, School of Public Health, University of Washington, , Seattle, WA, , United States

    8Firearm Injury and Policy Research Program, University of Washington, , Seattle, WA, , United States

    9Department of Emergency Medicine, University of Colorado School of Medicine, , Aurora, CO, , United States

    Corresponding Author:

    Julie Angerhofer Richards, MPH, PhD


    Background: “Lock to Live” (L2L) is a novel web-based decision aid for helping people at risk of suicide reduce access to firearms. Researchers have demonstrated that L2L is feasible to use and acceptable to patients, but little is known about how to implement L2L during web-based mental health care and in-person contact with clinicians.

    Objective: The goal of this project was to support the implementation and evaluation of L2L during routine primary care and mental health specialty web-based and in-person encounters.

    Methods: The L2L implementation and evaluation took place at Kaiser Permanente Washington (KPWA)—a large, regional, nonprofit health care system. Three dimensions from the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) model—Reach, Adoption, and Implementation—were selected to inform and evaluate the implementation of L2L at KPWA (January 1, 2020, to December 31, 2021). Electronic health record (EHR) data were used to purposefully recruit adult patients, including firearm owners and patients reporting suicidality, to participate in semistructured interviews. Interview themes were used to facilitate L2L implementation and inform subsequent semistructured interviews with clinicians responsible for suicide risk mitigation. Audio-recorded interviews were conducted via the web, transcribed, and coded, using a rapid qualitative inquiry approach. A descriptive analysis of EHR data was performed to summarize L2L reach and adoption among patients identified at high risk of suicide.

    Results: The initial implementation consisted of updates for clinicians to add a URL and QR code referencing L2L to the safety planning EHR templates. Recommendations about introducing L2L were subsequently derived from the thematic analysis of semistructured interviews with patients (n=36), which included (1) “have an open conversation,” (2) “validate their situation,” (3) “share what to expect,” (4) “make it accessible and memorable,” and (5) “walk through the tool.” Clinicians’ interviews (n=30) showed a strong preference to have L2L included by default in the EHR-based safety planning template (in contrast to adding it manually). During the 2-year observation period, 2739 patients reported prior-month suicide attempt planning or intent and had a documented safety plan during the study period, including 745 (27.2%) who also received L2L. Over four 6-month subperiods of the observation period, L2L adoption rates increased substantially from 2% to 29% among primary care clinicians and from <1% to 48% among mental health clinicians.

    Conclusions: Understanding the value of L2L from users’ perspectives was essential for facilitating implementation and increasing patient reach and clinician adoption. Incorporating L2L into the existing system-level, EHR-based safety plan template reduced the effort to use L2L and was likely the most impactful implementation strategy. As rising suicide rates galvanize the urgency of prevention, the findings from this project, including L2L implementation tools and strategies, will support efforts to promote safety for suicide prevention in health care nationwide.

    JMIR Med Inform 2024;12:e48007

    doi:10.2196/48007

    Keywords

    suicide prevention; firearm; internet; implementation; suicide; prevention; decision aid; risk; feasible; support; evaluation; mental health; electronic health record; tool 


    Firearm-related suicide accounts for approximately half of the suicide deaths in the United States annually [1]. Firearms are common in Americans’ lives [2]; about one-third of Americans report owning firearms [3], and an additional 10% report living in a household with a firearm [4], with higher rates in western states [2], among veterans [5], and in rural areas [6]. Moreover, the rate of ownership of new firearms appears to have increased recently among women, Black people, and Hispanic people [7]. Suicide attempts by firearm are highly lethal; researchers estimate that 85% to 95% of individuals who attempt suicide by firearm do not survive [8,9], and people with access to firearms, particularly if firearms are kept loaded and unlocked [10,11], have increased suicide risk [12,13]. Clinicians may have opportunities to intervene with patients at risk for firearm-related suicide because about 50% of individuals who die by suicide see a clinician in the month before death, and over 80% see one in the year before death [14]. Moreover, clinician-initiated discussions about reducing access to firearms have demonstrated effectiveness for improving firearm security practices (particularly in combination with free safe storage devices) [15-17], as well as promising findings for reducing suicide attempts [18,19].

    Despite its potential benefits, clinician-initiated dialogue about limiting access to firearms is an uncommon practice across many primary care and mental health specialty practices [18,20]. Common barriers include time, clinicians’ unfamiliarity with firearms, and concerns about negatively impacting relationships or alienating patients [21]. “Lock to Live” (L2L) is a self-directed, anonymous, web-based decision aid that was designed to address these barriers. L2L was developed in collaboration with clinicians, firearm owners, and people who had experienced suicidal thoughts and attempts [22]. Consistent with international design standards [23,24], the L2L decision aid steps users through various considerations regarding in-home and out-of-home firearm storage options, such as types of storage, costs of storage, and background check requirements, with a goal of encouraging storage solution discussions that are consistent with the users’ values and preferences [22]. Two subsequent research studies demonstrated promising results for the feasibility and acceptability of offering L2L emergency care encounters [25] and for the uptake of L2L when it was offered via secure patient portal messages after outpatient care encounters [26]. Though L2L appears to be a useful tool for supporting suicide prevention in clinical practice, little is known about how to use L2L during routine health care encounters outside of a research context.

    The goal of this project was to use mixed methods (qualitative and statistical evaluations) to support the implementation and evaluation of L2L during primary care and mental health specialty encounters in a large, regional health care system. Specifically, this project used semistructured interviews with clinicians and patients to support implementation, as well as statistical analyses to evaluate the reach and adoption of L2L over a 2-year period. The evaluation findings will inform considerations for implementing L2L nationwide to support suicide prevention in health care systems.


    Setting

    L2L implementation and evaluation took place at Kaiser Permanente Washington (KPWA)—1 of 8 regional Kaiser Permanente health care systems, which together form one of the nation’s largest nonprofit health care organizations and serve 12.5 million people [27]. At the time of this evaluation, KPWA had provided comprehensive medical care to approximately 700,000 members across Washington State via employer-sponsored insurance plans, individual insurance plans, or capitated Medicaid or Medicare programs. In 2016, KPWA augmented standard clinical workflows to support the identification and engagement of patients at high risk of suicide attempts (Figure 1) [28,29]. Specifically, a system-level electronic health record (EHR) template was created to support clinician-initiated safety planning among patients who are identified as at high risk of suicide during primary care and mental health specialty encounters [28,30]. Nationally, safety planning is a widely recommended best practice [31], and KPWA had an established process for safety planning that included addressing access to lethal means but did not offer any specific resources to clinicians or patients about firearm storage options. Consistent with the goal of L2L, step 6 of this safety plan template was designed to support patients in limiting access to lethal means, such as firearms and prescription medications.

    ‎
    Figure 1. Clinical workflow for supporting the identification and engagement of patients at high risk of suicide during primary and mental health specialty encounters at Kaiser Permanente Washington.

    Implementation and Evaluation Framework, Data Sources, and Study Design

    Three dimensions from the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) model—Reach, Adoption, and Implementation [32,33]—were selected to both inform and evaluate the implementation of L2L at KPWA (Multimedia Appendix 1) over a 2-year observation period (January 1, 2020, to December 31, 2021). Specifically, a qualitative, team-based, formative evaluation [34] was used, involving semistructured interviews with purposefully sampled patients and clinicians to facilitate implementation tools and strategies. Descriptive statistical analyses were used to evaluate the reach of L2L among patients identified at high risk of suicide and L2L adoption over a 2-year observation period. The findings were stratified by primary care and mental health specialty service settings due to the variation in the timing of the L2L trainings across these settings (Figure 2).

    ‎
    Figure 2. L2L implementation tools and strategies used over four 6-month subperiods of the 2-year observation period (January 1, 2020, to December 31, 2021). Tools are shown in blue, strategies are shown in green, and capped lines indicate semistructured interviews. EHR: electronic health record; KPWA: Kaiser Permanente Washington; L2L: Lock to Live; LICSW: licensed clinical social worker.

    Semistructured Qualitative Interviews

    EHR data were used to purposefully recruit adult patients to participate in semistructured interviews that included questions to elicit suggestions about introducing L2L, as part of a broader interview that focused on exploring perceptions regarding and experiences with firearm access assessment [35]. An invitation letter was mailed to sampled patients (age≥18 y) who had received a standardized question about firearm access (“Do you have access to guns? Yes/No”) in the prior 2 weeks on a mental health questionnaire [30]. A stratified sampling distribution was used to recruit approximately equal numbers of patients in 3 groups, including those who (1) reported firearm access, (2) reported no firearm access, and (3) did not respond (ie, left the question blank). The three sampling groups were also designed to purposefully include patients who had reported thoughts about self-harm in the prior 2 weeks via the ninth question of the 9-item Patient Health Questionnaire (PHQ-9) [36]. Interviewers attempted to reach all invitees for 2 weeks to invite them to participate in a phone interview. Following the portion of the interview guide that focused on firearm access assessment [35], interviewers described how patients reporting suicidal thoughts would soon be receiving L2L and elicited feedback about how to introduce this tool in a way that would make patients more likely to use it (Multimedia Appendix 2). This portion of the interview transcript was extracted into Excel (Microsoft Corporation) and analyzed by using a team-based, rapid, qualitative inquiry approach [37]. This involved an iterative data analysis wherein 2 coders independently coded the L2L portion of the transcript by using a combination of deductive and inductive content analyses, with codes developed a priori from the interview guide as well as codes that emerged from the interviews [38]. This was followed by several rounds of discussions with 2 additional team members who reviewed the coded data and reconciled themes iteratively for the purpose of using the summarized themes to facilitate implementation.

    Following the completion of the qualitative analysis that focused on patient-informed L2L implementation, interviewers initiated clinician recruitment activities, which were also more broadly focused on firearm access assessment. At the recommendation of care delivery leaders, interviewers outreached to the following two groups of clinicians responsible for engaging patients identified at risk of suicide in risk mitigation (ie, safety planning [39]): (1) licensed clinical social workers (LICSWs) supporting integrated mental health in primary care [28] and (2) consulting nurses (registered nurses) responsible for connecting patients (ie, those reporting suicidality after business hours via telephone) to telephone-based follow-up care. The L2L portion of the interview guide included questions informed by patient interviews (Multimedia Appendix 2) and was analyzed by using the same rapid, qualitative inquiry approach [37] that was used for patient interviews for the purpose of further facilitating L2L implementation.

    Descriptive Statistical Analyses

    L2L Reach and Adoption

    EHR data were used to summarize L2L “reach,” which was defined as the proportion of patients identified as at high risk of suicide via routine screening and assessment clinical workflows (Figure 1) and received the web-based decision aid. Specifically, we described characteristics of patients who had a documented safety plan during the 2-year observation period and characteristics of patients who had a safety plan that included a reference to L2L. Next, we described the adoption of L2L by primary care and mental health specialty clinicians over four 6-month subperiods of the observation period by calculating the proportions of patients identified at high risk of suicide (via suicide risk assessment; described in the Measures section) who had a documented safety plan with a reference to L2L and those who had a documented safety plan without a reference to L2L. We selected 6-month subperiods as the most helpful way to visually describe L2L adoption over time, since implementation paused during the initial COVID-19 outbreak (described in the Implementation Timeline, Tools, and Strategies section). We stratified by service setting due the variation in the timing of L2L trainings for these groups of clinicians.

    Measures

    The Columbia Suicide Severity Rating Scale (C-SSRS) was used to measure suicide risk, as per current clinical workflows. Specifically, patients reporting some level of suicide attempt planning or intent in the past month (ie, answering “yes” to C-SSRS question 3 or higher) were considered to be at “high risk” and alerted clinicians (via EHR prompts) to initiate safety planning. Distinctive phrases from standard EHR-based templates were used to detect safety plans documented in the text of clinical notes among patients identified at high risk (Multimedia Appendix 3). Sociodemographic and clinical characteristics of interest, including those known to be associated with firearm ownership and suicide risk [40,41], were measured by using the following administrative and diagnostic EHR data: age (continuous); sex (male or female); race and ethnicity (Asian, Black, Hispanic or Latinx, White, other, or unknown); insurance type (commercial, Medicare, Medicaid, or other); rurality (urban, large suburban, small suburban, or mostly rural) [42,43]; and prior year mental health, substance use, and self-harm diagnoses derived from the International Classification of Diseases, Tenth Revision, Clinical Modification. Reported firearm access was measured based on a positive response to the question on the mental health questionnaire [30] that was also used for qualitative interview recruitment (described in the Semistructured Qualitative Interviews section).

    Ethical Considerations

    The project team received approval from the KPWA Region Institutional Review Board (review number: 1826198) to conduct this study. Patients who agreed to participate in the phone interview provided oral consent, including permission for the interview to be audio-recorded and professionally transcribed, and they received a US $50 cash incentive for participation. During clinician recruitment activities, clinicians received up to 3 email invitations, which included a study information sheet and instructions for opting out of participation and further contact. Participating clinicians verbally consented to participation and received a US $50 gift card for participation.


    Implementation Timeline, Tools, and Strategies

    Over the 2-year implementation period (January 1, 2020, to December 31, 2021), a team of researchers and care delivery leaders took a pragmatic approach to iteratively creating and refining L2L implementation tools and strategies for primary care and mental health specialty clinicians (Figure 2). Initially, tools included an EHR-based macro (ie, EPIC SmartPhrase [Epic Systems Corporation]) for clinicians to easily add a URL and QR code referencing L2L to safety planning templates and a 1-page quick-start guide (ie, “Huddlecard”) with information on how to use the new SmartPhrase during routine clinic meetings (ie, “huddles”). In February 2020, the LICSWs who supported mental health care delivery in primary care [28] received information about L2L during a brief, web-based staff training session. Additional trainings that were planned for mental health specialty clinicians were put on hold during the widespread service disruption that subsequently occurred in response to the initial COVID-19 outbreak in March 2020. Additional tools and strategies were used, following recommendations from the care delivery leaders responsible for primary care and mental health service recovery and from the patients and clinicians who participated in semistructured qualitative interviews (detailed in the Findings From Semistructured Qualitative Interviews section).

    Findings From Semistructured Qualitative Interviews

    Of 76 patients who were purposefully sampled during 2 waves of recruitment, 36 were interviewed from November 18, 2019, to February 10, 2020 (Table 1). Five organizing themes were derived from the portion of the interview that elicited perceptions and suggestions about L2L and were used to create a handout for clinicians, with suggestions about how to introduce L2L to their patients at risk of suicide (Multimedia Appendix 4), including recommendations to (1) “have an open conversation,” (2) “validate their situation,” (3) “share what to expect,” (4) “make it accessible and memorable,” and (5) “walk through the tool” (Table 2). In addition to these recommendations, patients expressed a preference for receiving information about L2L from “trusted” and “caring” clinicians.

    Table 1. Characteristics of patient (n=36) and clinician (n=30) semistructured interview participants.
    PatientsClinicians
    Sex, n (%)
    Female17 (47)24 (80)
    Male19 (53)6 (20)
    Age (y), mean (SD)47.3 (17.9)44.3 (12.1)
    Age category (y), n (%)
    19-298 (22)1 (3)
    30-4911 (31)20 (67)
    50-649 (25)6 (20)
    ≥658 (22)3 (10)
    Race and ethnicity, n (%)
    American Indian or Alaska Native0 (0)1 (3)
    Black3 (8)2 (7)
    Asian or Pacific Islander3 (8)5 (17)
    Latinx or Hispanic1 (3)4 (13)
    Unknown2 (6)0 (0)
    White27 (75)18 (60)
    Reported firearm accessa,b, n (%)16 (44)N/Ac
    Reported thoughts about self-harm (prior 2 wk)a,d, n (%)15 (42)N/A

    aPatients’ responses recorded on the Kaiser Permanente Washington mental health monitoring questionnaire used for criterion sampling within the 2 wk prior to the recruitment initiation.

    b“Do you have access to guns? Yes/No.”

    cN/A: not applicable.

    dNinth question on the 9-item Patient Health Questionnaire: “Thoughts that you would be better off dead, or of hurting yourself.”

    Table 2. Thematic analysis of semistructured interviews with patients (n=36) and recommendations for introducing “Lock to Live” (L2L).
    ThemesIllustrative quotesaRecommendation
    Show caring and compassion, ask permission, and respect autonomy
    • “I think it’s important to just take a breath, sit down with them, hold their hand, look them in the eye - ‘how can I help you? Help me help you. What’s going on? Tell me. What are you thinking? How are you feeling? How can I help you?’ Instead of an assembly line and ‘I only have a few minutes,’ so they [providers] don’t take the time.” (Patient B029)
    • “I would hope the provider is very warm and caring and explains it’s a safety precaution, it’s for your better health and it insures you’ll be safer…basically it’s another part of your little toolbox to keep yourself well.” (Patient A032)
    • “Probably compassionately, potentially generalized to begin with, to find out if the person is resistant right upfront… Maybe you could put a question, ‘would you be willing to consider options for storing or access to lethal means, whether it’s firearms or medication? Is it something you would be willing to discuss and look into if you were experiencing suicidal thoughts?’” (Patient A005)
    • “We’re offering you the means to protect yourself, this is not an us decision, this is a you decision. So here is the website, the online information and we encourage you to look at it, but it’s your decision….Nobody can force you to do things. So bringing it up more as like – not we’re taking it [firearm] away from you, but letting you decide what to do with it.” (Patient A024)
    • “Reassuring people that their firearm ownership will not end because they’re going through a rough patch in life; their ability to have their own authority to hold onto their possessions [will not end], firearms or not.” (Patient B036)
    		“Have an open conversation”: patients were more willing to listen and try a tool if a clinician took the time to connect, showed compassion for people’s unique experiences, and showed respect for autonomy.
    Frame as helpful resource and normalize experiences
    • “I think overall education about the topic to start out with, just to say…‘this is what we have found is helpful, in these situations.’ Moving more into letting people know what their resources could be. Just education to be begin with, so it’s not so threatening. I think anytime somebody’s in a vulnerable place emotionally, they’re already possibly feeling threatened and they may not want to trust a lot of people.” (Patient A005)
    • “I would hope it would be pretty real, like a conversation…‘based on your health concerns you’re showing, we’ve got some important information we’d like to share with you,’ especially if that person has a relationship or feels responsible with the person presenting it, would stay there together and talk about it afterwards….Also statistics to help a person realize how more common this is.” (Patient A011)
    		“Validate their situation”: normalize their experience, share how common suicidal thoughts are, and be nonjudgmental in your approach; people have a variety of gun beliefs.
    Address privacy and security
    • “[The provider] would have to explain what it does, how it’s going to work and how private it is - nobody can get into your part of the website anyway, your personal page, where you go. So she has to reassure them about that….I just don’t feel that being online is that secure.” (Patient B004)
    • “privacy is probably number one and an assurance that you’re not being turned in….I would be concerned in our surveillance state that disclosing things to a website about my firearm use might somehow come close to violating some kind of civil right to privacy.” (Patient B008)
    • “As long as people don’t have to put in information which can be tracked, I can see lots of people using it. I mean the minute [you have to enter] your name, address, phone number, medical ID number, whatever else, people are going to go – eh.” (Patient A033)
    		“Share what to expect”: address privacy and how information is stored if patients visit the website; assure patients that L2L is anonymous.
    Accessibility is key
    • “You don’t want to make it hard to find on a website because it doesn’t take me very long. If something’s really hard to find on a website, I’m out of there.” (Patient B004)
    • “If there are hoops to jump through before you can access it, if you have to log in, go through a bunch of pages - maybe if it was right there, ready to access at any time, I’d say that’d be better.” (Patient B008)
    • “I’d love to see it everywhere. Have little cards that could be given out, a billboard, having my doctor [send it].” (Patient A033)
    • “If I knew it existed, I would probably try it. advertise it.” (Patient A032)
    • “Highlight it in your After Visit Summary too.” (Patient A002)
    		“Make it accessible and memorable”: have multiple routes for sharing the website and sending reminders (after-visit summary, message, website, pamphlet).
    Demonstrate and “show, don’t just tell”
    • “I think when you’re in a pit of despair, to go and do it on your own, some people will do that and other people will not. They need to be taken by the hand and go, ‘what do you think about this?’ Read it together.” (Patient B036)
    • “I’m more keen to follow somebody who’s like ‘I’m offering you the opportunity to maybe do this together,’ instead of ‘I’m watching out for you.’” (Patient A024)
    • “Being shown an example would be nice, showing it off briefly. Knowing more specifically what it does or how it could be helpful as opposed to just knowing it exists.” (Patient B033)
    • “I think showing the patient or at least offering, would you like me to show you? Not just telling somebody, because short term memory is only like 30 s or a couple minutes and then you forget about it.” (Patient A002)
    		“Walk through the tool”: most patients said that a website walk-through, rather than simply having a conversation, would be helpful to overcome the barrier of trying something new, especially if already depressed.

    aIdentifier “A”: patients in the first wave of interviews; identifier “B”: patients in the second wave of interviews (grammatic edits, noted in brackets, were added to clarify intended meaning).

    Of 51 purposefully sampled clinicians responsible for safety planning with patients identified at high risk of suicide, 30 were interviewed from July 7, 2020, to October 8, 2020 (Table 1), including 25 LICSWs and 5 registered nurses. During the interviews with LICSWs, only 3 had actually used L2L with a patient—9 were unfamiliar with L2L, and 12 were familiar with but had not yet used L2L. Most clinicians saw clear benefits to L2L as an option for supporting both clinicians and patients. Several clinicians expressed concern about using the tool to replace dialogue about lethal means, and most supported the idea of a walk-through, as patients had recommended. Clinicians also expressed a strong preference to have L2L information included by default in the EHR-based safety planning template, in contrast to having clinicians remember to add it (via SmartPhrase). A clinician also suggested automatically including L2L in after-visit summaries when patients reported thoughts about self-harm on the PHQ-9. The implementation team worked with clinical partners to update the system-level, EHR-based safety plan template to include L2L information and updated the Huddlecard to communicate this change (Multimedia Appendix 5). After-visit summaries were used to provide safety plans to patients who were seen via a secure, web-based patient portal, and L2L was automatically included after the template change. Several clinicians also requested follow-up trainings or refreshers about L2L. The team therefore conducted a round of brief trainings, which were presented during routine clinic huddles with mental health specialty clinicians, and created a 3-minute training video (Multimedia Appendix 6).

    Findings From Descriptive Statistical Analyses

    During the study period, 2739 adult patients reported some prior-month suicide attempt planning or intent via routine suicide risk assessment workflows during primary care or mental health specialty encounters and had a documented safety plan, including 745 (27.2%) who also received L2L. Overall, there were no major differences in the demographic and clinical characteristics between patients who received L2L and the broader population that was identified as at risk of suicide and had a documented safety plan (Table 3).

    Table 3. Characteristics of patients who received “Lock to Live” (L2L; n=745) and were among patients with a documented safety plan (n=2739) during the implementation period (January 1, 2020, to December 31, 2021).
    Patients who received L2L, n (%)Patients with a documented safety plana, n (%)
    Ageb (y)
    18-39513 (68.9)1817 (66.3)
    40-64187 (25.1)732 (26.7)
    ≥6545 (6)190 (6.9)
    Sexc
    Female445 (59.7)1753 (64)
    Male300 (40.3)986 (36)
    Race and ethnicityc
    American Indian or Alaska Native14 (1.9)75 (2.7)
    Asian54 (7.2)199 (7.3)
    Black52 (7)166 (6.1)
    Hawaiian or Pacific Islander8 (1.1)45 (1.6)
    Hispanic or Latinx59 (7.9)201 (7.3)
    Unknown93 (12.5)294 (10.7)
    White465 (62.4)1759 (64.2)
    Insurancec
    Commercial530 (71.1)1831 (66.8)
    Medicare83 (11.1)342 (12.5)
    Medicaid54 (7.2)228 (8.3)
    Not enrolled78 (10.5)338 (12.3)
    Rural or urbanc,d
    Urban301 (40.4)1010 (36.9)
    Large suburban205 (27.5)799 (29.2)
    Smaller suburban186 (25)802 (29.3)
    Mostly rural31 (4.2)96 (3.5)
    Mental health diagnosese
    Depression675 (90.6)2502 (91.3)
    Anxiety652 (87.5)2434 (88.9)
    Serious mental illness144 (19.3)586 (21.4)
    Substance use disorder196 (26.3)752 (27.5)
    Suicide attempt diagnosis39 (5.2)175 (6.4)
    Reported firearm accesse150 (20.1)501 (18.3)

    aIncludes safety plans with L2L.

    bAt evaluation midpoint (January 1, 2021).

    cAt first encounter.

    dMissing information for 22 patients.

    eDuring implementation period.

    The adoption of L2L increased substantially over the 2-year observation period (Tables 4 and 5). During this time, rates of documented safety plans among patients identified at high risk of suicide (C-SSRS score≥3) remained fairly consistent—51.2% to 55.2% of primary care patients and 73.4% to 78.4% of mental health specialty patients had a documented safety plan. However, over four 6-month subperiods of the observation period, L2L adoption rates increased substantially from 2% to 29% among primary care clinicians and <1% to 48% among mental health clinicians, increasing primarily after L2L was integrated into the EHR-based safety planning template.

    Table 4. Proportions of primary care patients who were identified as at high risk of suicide and had a documented safety plan during primary care encounters over the implementation period (January 1, 2020, to December 31, 2021).
    Subperiods of implementation periodPatients with a documented safety plan that did not include “Lock to Live,” %Patients with a documented safety plan that did include “Lock to Live,” %
    Months 1-653.51.6
    Months 7-12504.1
    Months 13-1841.711.1
    Months 19-2422.229.1
    Table 5. Proportions of primary care patients who were identified as at high risk of suicide and had a documented safety plan during mental health specialty encounters over the implementation period (January 1, 2020, to December 31, 2021).
    Subperiods of implementation periodPatients with a documented safety plan that did not include “Lock to Live,” %Patients with a documented safety plan that did include “Lock to Live,” %
    Months 1-678.10.3
    Months 7-1274.81.4
    Months 13-1853.819.6
    Months 19-2425.948.4

    This novel study used mixed methods to support the implementation and evaluation of a web-based decision aid that was designed to help patients at risk of suicide limit access to firearms. Specifically, findings from semistructured interviews with patients and clinicians were used to facilitate L2L implementation, while statistical analyses were used to describe rates of reach among patients identified at risk of suicide and increased adoption by clinicians who cared for them during the 2-year observation period.

    L2L development centered users’ values and preferences in the design process [22]. Similarly, the tools and strategies developed for this project used information from semistructured interviews with people who were the most likely to be impacted by L2L implementation, including firearm owners, patients experiencing suicidality, and the clinicians who care for them. Clinicians have reported a lack of experience with handling firearms and have expressed apprehension about discussing firearm safety due to concerns about damaging relationships with patients [44-46]. Likewise, patients have expressed apprehension about disclosing access to firearms due to concerns about privacy, autonomy, and firearm ownership rights [47,48]. For these reasons, patients and clinicians perceive firearm access assessment as challenging but also as valuable for supporting suicide prevention [35]. This implementation project showed that clinicians, that is, those responsible for engaging at-risk primary care and mental health patients in suicide risk mitigation, willingly adopted the use of L2L to support safety planning.

    This study also has important implementation implications. Unsurprisingly, the rates of L2L adoption increased after L2L was incorporated into the existing system-level safety planning template as a default (primarily in the latter half of year 2). This finding underscores the importance of removing barriers to the adoption of web-based decision aids and making adoption “easy” [49]. In contrast, those seeking change often focus on amplifying benefits or “selling” their new idea or innovation; however, it may be equally as important or more important to focus on “friction,” that is, “psychological forces that oppose and undermine change,” such as inertia, effort, emotion, and reactance [50]. In the case of L2L, reducing the effort required for clinicians to remember to use L2L appeared to be the main driver of its adoption. However, the tools and strategies that were designed to communicate about the benefits of using L2L (eg, training, video, Huddlecard, and newsletter information) were likely necessary for leaders to understand L2L’s value to patients and clinicians and approve the system-level change that was required to make L2L easier to use for clinicians.

    This study has important clinical implications for supporting suicide prevention in health care. First, L2L supports clinicians who engage patients identified at risk of suicide in collaborative safety planning and lethal means counseling, which are evidence-based suicide risk mitigation practices that are recommended by the Zero Suicide Institute and follow the principles outlined in the National Strategy for Suicide Prevention [29,51-53]. Moreover, the recommendations from interview participants (“have an open conversation,” “share what to expect,” and “walk through the tool”) support a motivational interviewing approach to lethal means counseling and align with the recommendations of the Veterans Health Administration [54]. Second, L2L was developed by patients with lived experiences of suicidality and firearm ownership; therefore, L2L supports cultural competency in health care as a culturally aligned intervention [55]. Finally, this technology-based, EHR-embedded approach to addressing lethal means supports all 6 aims of health care quality that are outlined by the Institute of Medicine—safe, effective, patient-centered, timely, efficient, and equitable [56,57].

    There are several limitations of this project that have implications for future research. First, the implementation of L2L at KPWA occurred during the initial outbreak of a global pandemic, which impacted the original implementation plans while health care systems responded to the pandemic and rapidly shifted toward providing web-based mental health care [58]. Semistructured interviews with patients took place prior to this shift. Future research should explore optimizing mental health care delivery workflows that support web-based suicide risk identification (ie, screening and assessment) [59] and incorporating L2L in web-based care encounters via secure telehealth platforms that are designed to support patient engagement. Second, L2L recognizes and addresses firearm policies related to background checks and how these policies might influence the legality of temporary firearm transfers for addressing suicide risk, but it does not address specific state laws. Additional work to understand the legality of recommendations about firearm safety practices may be helpful for health care systems that implement L2L. Third, this project was not designed to measure the specific impact of individual implementation strategies or determine whether L2L was effective in helping patients reduce access to firearms for suicide prevention purposes. Measuring the effectiveness of this tool, which was designed to support population-based suicide prevention, would require extending the implementation of L2L to other large health care systems nationwide and conducting other analyses that are designed to measure key functions of suicide prevention practices, including risk identification, engagement in evidence-based risk mitigation and treatment, and supportive care transitions [29]. Finally, L2L is meant to support adult patients at risk of suicide reduce access to firearms and other lethal means; additional tools and strategies are required to support youth at risk of suicide. Notably, there is a similar web-based decision aid that is available for this purpose; “Lock and Protect” was designed to help parents and caregivers reduce access to lethal means for youth suicide risk mitigation [60]. Similarly, the “Safety in Dementia” web-based decision aid was developed to support caregivers in addressing firearm access among individuals with Alzheimer disease and related dementias [61]. Future research should evaluate the implementation of these tools in routine care delivery.

    In conclusion, incorporating L2L into the existing system-level safety plan template reduced the effort required to use L2L and was likely the most impactful implementation strategy for increasing clinician adoption and patient reach. However, understanding the value of L2L from the users’ perspectives was essential for effectively amplifying the suicide risk mitigation benefits. As rising suicide rates galvanize the urgency of prevention [62], the implementation tools and strategies developed for this project will be useful for health care systems nationwide.

    Acknowledgments

    This implementation evaluation was funded by Kaiser Permanente’s Office of Community Health and Center for Gun Violence Research and Education, as part of its Firearm Injury Prevention Program, and the Centers for Disease Control and Prevention (R01CE003460). The views and opinions expressed in this article are the responsibility of the authors and do not necessarily represent the official views of Kaiser Permanente or the Centers for Disease Control and Prevention. The authors acknowledge that this research would not be possible without the people who receive health care from Kaiser Permanente Washington and all the clinicians and staff who support the organization.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    The three dimensions of RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) selected to inform and evaluate the implementation of “Lock to Live.”

    DOCX File, 29 KB
    Multimedia Appendix 2

    Patient and clinician interview questions that focused on “Lock to Live” implementation.

    DOCX File, 33 KB
    Multimedia Appendix 3

    Elements included in the search for safety plans documented in clinical notes text.

    DOCX File, 31 KB
    Multimedia Appendix 4

    Introducing Lock2Live.org: a guide for clinicians.

    DOCX File, 295 KB
    Multimedia Appendix 5

    “Lock to Live” Huddlecard.

    DOCX File, 589 KB
    Multimedia Appendix 6

    “Lock to Live” video for Kaiser Permanente Washington clinicians.

    MP4 File, 13608 KB
    1. WISQARS — your source for U.S. injury statistics. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/injury/wisqars/facts.html [Accessed 2022-04-26]
    2. McCracken H, Okuley H, Floyd L. Gun ownership in America. RAND Corporation. URL: https://www.rand.org/research/gun-policy/gun-ownership.html [Accessed 2022-01-10]
    3. Kalesan B, Villarreal MD, Keyes KM, Galea S. Gun ownership and social gun culture. Inj Prev. Jun 2016;22(3):216-220. [CrossRef] [Medline]
    4. Parker K, Horowitz JM, Igielnik R, Oliphant JB, Brown A. America’s complex relationship with guns. Pew Research Center. Jun 22, 2017. URL: https://www.pewresearch.org/social-trends/2017/06/22/americas-complex-relationship-with-guns/ [Accessed 2022-03-29]
    5. Cleveland EC, Azrael D, Simonetti JA, Miller M. Firearm ownership among American veterans: findings from the 2015 National Firearm Survey. Inj Epidemiol. Dec 19, 2017;4(1). [CrossRef] [Medline]
    6. Nordstrom DL, Zwerling C, Stromquist AM, Burmeister LF, Merchant JA. Rural population survey of behavioral and demographic risk factors for loaded firearms. Inj Prev. Jun 2001;7(2):112-116. [CrossRef] [Medline]
    7. Miller M, Zhang W, Azrael D. Firearm purchasing during the COVID-19 pandemic: results from the 2021 National Firearms Survey. Ann Intern Med. Feb 2022;175(2):219-225. [CrossRef] [Medline]
    8. Anestis MD. Guns and Suicide: An American Epidemic. Oxford University Press; 2018.
    9. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. Fatal Injury Reports, National, Regional and State, 1981-2017, Web-Based Injury Statistics Query and Reporting System (WISQARS). Centers for Disease Control and Prevention; 2019.
    10. Kellermann AL, Rivara FP, Somes G, et al. Suicide in the home in relation to gun ownership. N Engl J Med. Aug 13, 1992;327(7):467-472. [CrossRef] [Medline]
    11. Shenassa ED, Rogers ML, Spalding KL, Roberts MB. Safer storage of firearms at home and risk of suicide: a study of protective factors in a nationally representative sample. J Epidemiol Community Health. Oct 2004;58(10):841-848. [CrossRef] [Medline]
    12. Anglemyer A, Horvath T, Rutherford G. The accessibility of firearms and risk for suicide and homicide victimization among household members: a systematic review and meta-analysis. Ann Intern Med. Jan 21, 2014;160(2):101-110. [CrossRef] [Medline]
    13. Mann JJ, Michel CA. Prevention of firearm suicide in the United States: what works and what is possible. Am J Psychiatry. Oct 1, 2016;173(10):969-979. [CrossRef] [Medline]
    14. Ahmedani BK, Simon GE, Stewart C, et al. Health care contacts in the year before suicide death. J Gen Intern Med. Jun 2014;29(6):870-877. [CrossRef] [Medline]
    15. Rowhani-Rahbar A, Simonetti JA, Rivara FP. Effectiveness of interventions to promote safe firearm storage. Epidemiol Rev. 2016;38(1):111-124. [CrossRef] [Medline]
    16. Simonetti JA, Rowhani-Rahbar A, King C, Bennett E, Rivara FP. Evaluation of a community-based safe firearm and ammunition storage intervention. Inj Prev. Jun 2018;24(3):218-223. [CrossRef] [Medline]
    17. Runyan CW, Becker A, Brandspigel S, Barber C, Trudeau A, Novins D. Lethal means counseling for parents of youth seeking emergency care for suicidality. West J Emerg Med. Jan 2016;17(1):8-14. [CrossRef] [Medline]
    18. Boggs JM, Beck A, Ritzwoller DP, Battaglia C, Anderson HD, Lindrooth RC. A quasi-experimental analysis of lethal means assessment and risk for subsequent suicide attempts and deaths. J Gen Intern Med. Jun 2020;35(6):1709-1714. [CrossRef] [Medline]
    19. Monuteaux MC, Azrael D, Miller M. Association of increased safe household firearm storage with firearm suicide and unintentional death among US youths. JAMA Pediatr. Jul 1, 2019;173(7):657-662. [CrossRef] [Medline]
    20. Boggs JM, Quintana LM, Powers JD, Hochberg S, Beck A. Frequency of clinicians’ assessments for access to lethal means in persons at risk for suicide. Arch Suicide Res. 2022;26(1):127-136. [CrossRef] [Medline]
    21. Walters H, Kulkarni M, Forman J, Roeder K, Travis J, Valenstein M. Feasibility and acceptability of interventions to delay gun access in VA mental health settings. Gen Hosp Psychiatry. 2012;34(6):692-698. [CrossRef] [Medline]
    22. Betz ME, Knoepke CE, Siry B, et al. ‘Lock to Live’: development of a firearm storage decision aid to enhance lethal means counselling and prevent suicide. Inj Prev. Sep 2019;25(Suppl 1):i18-i24. [CrossRef] [Medline]
    23. Stacey D, Bennett CL, Barry MJ, et al. Decision aids for people facing health treatment or screening decisions. Cochrane Database Syst Rev. Oct 5, 2011;10(10):CD001431. [CrossRef] [Medline]
    24. Légaré F, O’Connor AC, Graham I, et al. Supporting patients facing difficult health care decisions: use of the Ottawa Decision Support Framework. Can Fam Physician. Apr 2006;52(4):476-477. [Medline]
    25. Betz ME, Knoepke CE, Simpson S, et al. An interactive web-based lethal means safety decision aid for suicidal adults (Lock to Live): pilot randomized controlled trial. J Med Internet Res. Jan 29, 2020;22(1):e16253. [CrossRef] [Medline]
    26. Boggs JM, Quintana LM, Beck A, et al. "Lock to Live” for firearm and medication safety: feasibility and acceptability of a suicide prevention tool in a learning healthcare system. Front Digit Health. Sep 6, 2022;4:974153. [CrossRef] [Medline]
    27. Fast facts. Kaiser Permanente. URL: https://about.kaiserpermanente.org/who-we-are/fast-facts [Accessed 2021-11-08]
    28. Richards JE, Parrish R, Lee A, Bradley K, Caldeiro R. An integrated care approach to identifying and treating the suicidal person in primary care. Psychiatric Times. URL: https:/​/www.​psychiatrictimes.com/​view/​integrated-care-approach-identifying-and-treating-suicidal-person-primary-care [Accessed 2020-01-31]
    29. Richards JE, Simon GE, Boggs JM, et al. An implementation evaluation of “Zero Suicide” using normalization process theory to support high-quality care for patients at risk of suicide. Implement Res Pract. Jan 1, 2021;2:26334895211011769. [CrossRef] [Medline]
    30. Richards JE, Kuo E, Stewart C, et al. Self-reported access to firearms among patients receiving care for mental health and substance use. JAMA Health Forum. Aug 6, 2021;2(8):e211973. [CrossRef] [Medline]
    31. National Academies of Sciences, Engineering, and Medicine. Health Systems Interventions to Prevent Firearm Injuries and Death: Proceedings of a Workshop. The National Academies Press; 2019. [CrossRef]
    32. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. Sep 1999;89(9):1322-1327. [CrossRef] [Medline]
    33. Holtrop JS, Rabin BA, Glasgow RE. Qualitative approaches to use of the RE-AIM framework: rationale and methods. BMC Health Serv Res. Mar 13, 2018;18(1):177. [CrossRef] [Medline]
    34. Stetler CB, Legro MW, Wallace CM, et al. The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. Feb 2006;21(Suppl 2):S1-S8. [CrossRef] [Medline]
    35. Richards JE, Kuo ES, Whiteside U, et al. Patient and clinician perspectives of a standardized question about firearm access to support suicide prevention: a qualitative study. JAMA Health Forum. Nov 4, 2022;3(11):e224252. [CrossRef] [Medline]
    36. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med. Sep 2001;16(9):606-613. [CrossRef] [Medline]
    37. Beebe J. Rapid Qualitative Inquiry: A Field Guide to Team-Based Assessment. Rowman & Littlefield; 2014.
    38. Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. Nov 2005;15(9):1277-1288. [CrossRef] [Medline]
    39. Stanley B, Brown GK. Safety planning intervention: a brief intervention to mitigate suicide risk. Cogn Behav Pract. May 2012;19(2):256-264. [CrossRef]
    40. Nock MK, Borges G, Bromet EJ, Cha CB, Kessler RC, Lee S. Suicide and suicidal behavior. Epidemiol Rev. 2008;30(1):133-154. [CrossRef] [Medline]
    41. Morgan ER, Gomez A, Rowhani-Rahbar A. Firearm ownership, storage practices, and suicide risk factors in Washington State, 2013-2016. Am J Public Health. Jul 2018;108(7):882-888. [CrossRef] [Medline]
    42. Goldstick JE, Carter PM, Cunningham RM. Current epidemiological trends in firearm mortality in the United States. JAMA Psychiatry. Mar 1, 2021;78(3):241-242. [CrossRef] [Medline]
    43. Ingram DD, Franco SJ. 2013 NCHS Urban-Rural Classification Scheme for Counties. Vital Health Stat 2. Apr 2014;(166):1-73. [Medline]
    44. Ketterer AR, Poland S, Ray K, Abuhasira R, Aldeen AZ. Emergency providers’ familiarity with firearms: a national survey. Acad Emerg Med. Mar 2020;27(3):185-194. [CrossRef] [Medline]
    45. Farcy DA, Doria N, Moreno-Walton L, et al. Emergency physician survey on firearm injury prevention: where can we improve? West J Emerg Med. Feb 8, 2021;22(2):257-265. [CrossRef] [Medline]
    46. Wintemute GJ, Betz ME, Ranney ML. You can: physicians, patients, and firearms. Ann Intern Med. Aug 2, 2016;165(3):205-213. [CrossRef] [Medline]
    47. Richards JE, Hohl SD, Segal CD, et al. “What will happen if I say yes?” perspectives on a standardized firearm access question among adults with depressive symptoms. Psychiatr Serv. Aug 1, 2021;72(8):898-904. [CrossRef] [Medline]
    48. Khazanov GK, Keddem S, Hoskins K, et al. Stakeholder perceptions of lethal means safety counseling: a qualitative systematic review. Front Psychiatry. Oct 20, 2022;13:993415. [CrossRef] [Medline]
    49. Service O, Hallsworth M, Halpern D, et al. EAST: four simple ways to apply behavioural insights. The Behavioural Insights Team. URL: https://www.bi.team/wp-content/uploads/2015/07/BIT-Publication-EAST_FA_WEB.pdf [Accessed 2023-10-12]
    50. Nordgren L, Schonthal D. The Human Element: Overcoming the Resistance That Awaits New Ideas. Wiley; 2021.
    51. Zero Suicide Institute. Education Development Center. URL: https://solutions.edc.org/solutions/zero-suicide-institute [Accessed 2023-10-12]
    52. U.S. Department of Health and Human Services (HHS) Office of the Surgeon General and National Action Alliance for Suicide Prevention. 2012 National Strategy for Suicide Prevention: Goals and Objectives for Action. HHS; 2012.
    53. Layman DM, Kammer J, Leckman-Westin E, et al. The relationship between suicidal behaviors and Zero Suicide organizational best practices in outpatient mental health clinics. Psychiatr Serv. Oct 1, 2021;72(10):1118-1125. [CrossRef] [Medline]
    54. Lethal means safety counseling. US Department of Veterans Affairs. URL: https://www.mirecc.va.gov/visn19/lethalmeanssafety/counseling/ [Accessed 2023-10-12]
    55. TIP 59: improving cultural competence. Substance Abuse and Mental Health Services Administration. 2015. URL: https://store.samhsa.gov/product/TIP-59-Improving-Cultural-Competence/SMA15-4849 [Accessed 2023-10-12]
    56. Institute of Medicine (US) Committee on Quality of Health Care in America. Crossing the Quality Chasm: A New Health System for the 21st Century. National Academies Press; 2001. [CrossRef]
    57. Six domains of healthcare quality. Agency for Healthcare Research and Quality. Feb 2015. URL: https://www.ahrq.gov/talkingquality/measures/six-domains.html [Accessed 2024-03-20]
    58. Wosik J, Fudim M, Cameron B, et al. Telehealth transformation: COVID-19 and the rise of virtual care. J Am Med Inform Assoc. Jun 1, 2020;27(6):957-962. [CrossRef] [Medline]
    59. Simon GE, Stewart CC, Gary MC, Richards JE. Detecting and assessing suicide ideation during the COVID-19 pandemic. Jt Comm J Qual Patient Saf. Jul 2021;47(7):452-457. [CrossRef] [Medline]
    60. Asarnow JR, Zullo L, Ernestus SM, et al. “Lock and Protect”: development of a digital decision aid to support lethal means counseling in parents of suicidal youth. Front Psychiatry. Oct 6, 2021;12:736236. [CrossRef] [Medline]
    61. McCarthy V, Portz J, Fischer SM, et al. A web-based decision aid for caregivers of persons with dementia with firearm access (Safe at Home study): protocol for a randomized controlled trial. JMIR Res Protoc. Jan 31, 2023;12:e43702. [CrossRef] [Medline]
    62. Barry E. Following a two-year decline, suicide rates rose again in 2021. The New York Times. Feb 11, 2023. URL: https:/​/www.​nytimes.com/​2023/​02/​11/​health/​suicide-rates-cdc.​html#:~:text=Suicide%20increased%20among%20younger%20Black,reported [Accessed 2023-02-13]

    ‎
    C-SSRS: Columbia Suicide Severity Rating Scale
    EHR: electronic health record
    KPWA: Kaiser Permanente Washington
    L2L: Lock to Live
    LICSW: licensed clinical social worker
    PHQ-9: 9-item Patient Health Questionnaire
    RE-AIM: Reach, Effectiveness, Adoption, Implementation, Maintenance

    Edited by Jennifer Hefner; submitted 07.04.23; peer-reviewed by Gabriela Khazanov, Jeff Sung; final revised version received 12.10.23; accepted 27.02.24; published 22.04.24.

    Copyright

    © Julie Elissa Richards, Elena Kuo, Christine Stewart, Lisa Shulman, Rebecca Parrish, Ursula Whiteside, Jennifer M Boggs, Gregory E Simon, Ali Rowhani-Rahbar, Marian E Betz. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 22.4.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.