Published on in Vol 9, No 6 (2021): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/28497, first published .
Using Electronic Health Records to Mitigate Workplace Burnout Among Clinicians During the COVID-19 Pandemic: Field Study in Iran

Using Electronic Health Records to Mitigate Workplace Burnout Among Clinicians During the COVID-19 Pandemic: Field Study in Iran

Using Electronic Health Records to Mitigate Workplace Burnout Among Clinicians During the COVID-19 Pandemic: Field Study in Iran

Authors of this article:

Pouyan Esmaeilzadeh1 Author Orcid Image ;   Tala Mirzaei1 Author Orcid Image

Original Paper

Department of Information Systems and Business Analytics, College of Business, Florida International University, Miami, FL, United States

*all authors contributed equally

Corresponding Author:

Pouyan Esmaeilzadeh, PhD

Department of Information Systems and Business Analytics

College of Business

Florida International University

Modesto A Maidique Campus 11200 SW 8th St

RB 261 B

Miami, FL, 33199

United States

Phone: 1 3053483302

Email: pesmaeil@fiu.edu


Background: The COVID-19 pandemic spread worldwide in 2020. Notably, in the countries dealing with massive casualties, clinicians have worked in new conditions characterized by a heavy workload and a high risk of being infected. The issue of clinician burnout during the pandemic has attracted considerable attention in health care research. Electronic health records (EHRs) provide health care workers with several features to meet a health system’s clinical needs.

Objective: We aim to examine how the use of EHR features affects the burnout of clinicians working in hospitals that have special wards for confirmed COVID-19 cases.

Methods: Using an online survey, we collected data from 368 physicians, physician assistants, and nurses working in six hospitals that have implemented EHRs in the city of Tehran in Iran. We used logistic regression to assess the association between burnout and awareness of EHR features, EHR system usability, concerns about COVID-19, technology solutions, hospital technology interventions, hospital preparedness, and professional efficacy adjusted for demographic and practice characteristics.

Results: The primary outcome of our study was self-reported burnout during the COVID-19 pandemic. Of the 368 respondents, 36% (n=134) reported having at least one symptom of burnout. Participants indicated that the leading cause of EHR-related stress is inadequate training for using technology (n=159, 43%), followed by having less face-to-face time with patients (n=140, 38%). Positive perceptions about the EHR’s ease of use were associated with lower odds of burnout symptoms. More interventions, such as clear communication of regulations; transparency in policies, expectations, and goals regarding the use of technology in the clinical workflow; and hospital preparedness to cope with the challenges of the pandemic, were associated with lower odds of burnout.

Conclusions: The use of EHR applications, hospital pandemic preparation programs, and transparent technology-related policies and procedures throughout the epidemic can be substantial mitigators of technology-based stress and clinician burnout. Hospitals will then be better positioned to devise or modify technology-related policies and procedures to support physicians’ and nurses’ well-being during the COVID-19 pandemic. Training programs, transparency in communications of regulations, and developing a clear channel for informing clinicians of changes in policies may help reduce burnout symptoms among physicians and nurses during a pandemic. Providing easily accessible mentorship through teleconsultation and 24-hour available information technology support may also help to mitigate the odds of burnout.

JMIR Med Inform 2021;9(6):e28497

doi:10.2196/28497

Keywords



Burnout has attracted more attention in health care since it is considered a trigger for health care professionals’ physical and mental problems [1]. Clinicians’ burnout may negatively affect the quality of care, cost of health care delivery, productivity, and patient satisfaction [2]. Prior research indicates different organizational, relational, and work-related factors contributing to clinicians’ burnout [3,4]. COVID-19 was first identified in Wuhan, China and has since become a global pandemic that has spread to more than 150 countries and affected over 9.6 million people worldwide [5]. Recently with the spread of the COVID-19 pandemic, burnout syndrome and physical exhaustion among clinicians have become even more pronounced. Declaring the COVID-19 outbreak as a pandemic by the World Health Organization has raised concerns about its possible detrimental effects on clinicians’ workload and well-being [6].

During the outbreak, clinicians have been exposed to patients with severe or mild symptoms. Since respiratory droplets and close contact are the main transmitters of COVID-19, clinicians are at higher risk of being affected [7]. Besides the health-related stress, health care centers have had a sudden increase in phone calls, patient portal messages, appointment requests, ambulatory care visits, and walk-in patients [8]. Accommodating patients with lots of concerns, questions, and clinical demands has also increased health care professionals’ workload. These burnout symptoms are exhibited by not only health care professionals who are specialized in infectious diseases but also physicians with different specialties, and nurses may experience more significant work pressure dealing with suspected and confirmed cases [7]. Previous studies have provided evidence to indicate that clinicians have experienced stress, anxiety, and depression during the COVID-19 pandemic [9,10].

Research to date has identified several contributing factors associated with burnout, including health information technology (HIT). Electronic health record (EHR) systems, for example, are often seen as cumbersome to use, failing to fulfill the promise of improved health care delivery and little more than a means of meeting regulatory and billing requirements [11,12]. However, during the COVID-19 outbreak, health care organizations are establishing different strategies and leveraging several health information technologies to improve pandemic management [13]. One of the technology-based tools that can support the standard management of patients during the current pandemic is the EHR [14,15].

The EHR has some features to enable the use of standardized processes such as scripted triaging, immediate health information exchange, real-time data analytics, electronic check-in, self-screening pages, and telemedicine [16]. Providing remote care through telemedicine and teleconsulting for patients became more prevalent during the outbreak. Using patient portals to provide instructions and medication prescriptions increased as well. There remains considerable debate in the informatics community about the actual role health information technologies play in shaping clinician burnout during the pandemic. Existing research suggests that technologies may be confounded with other important causes, including changes in regulatory mandates, increased clinical volumes, and flexibility to mitigate ad hoc challenges. Regardless of the role information technology (IT) plays in clinician burnout, innovative solutions to alleviate burnout during the pandemic are urgently needed. A recent article explains that “EHR innovations cannot help to mitigate clinician burnout without careful consideration of the socioecological context in which these innovations occur, including organizational culture, the healthcare marketplace, technology ecosystem, and national policy” [17]. There is a lack of research investigating the impact of this pandemic on burnout among clinicians in resource-limited countries and the possible impact of HIT on clinicians’ stress and burnout [18].

We investigate how clinicians’ awareness of EHR features and perception of EHR system usability as well as their perceptions about the hospital policies and preparedness can help reduce the burnout caused during the pandemic for clinicians in Iran, which was one of the resource-limited countries that had been reported to be among the first 15 countries to have COVID-19 patients and deaths [19]. As of April 8, 2021, Iran has 1,984,348 total confirmed cases and 63,699 deaths [19]. EHRs are still developing in Iran, and various hospitals are at a different level of adoption and use of EHRs. Hospitals in Iran use several EHR applications and informatics infrastructure for outbreak management [14,20]. However, little is known about how the use of EHR capabilities and clinicians’ perceptions of this technology in the Iranian health system during the COVID-19 outbreak successfully reverse burnout among clinicians to manage this novel infection. This study’s findings can contribute insights to develop better strategies for using EHRs, refining policies, and enhancing preparedness to reduce the increasing rate of burnout symptoms during the pandemics in resource-limited countries.


Setting

There are 118 hospitals in Tehran, the capital city of Iran, and among them, only 15 hospitals have wards specialized for quarantining and treating patients with COVID-19 (at the time of this research). In this study, we considered six of these hospitals with large administrations. These six hospitals were mainly selected because they were among the first hospitals in Tehran that have been quarantining and treating patients with COVID-19. They encompass outpatient primary and specialty medical and surgical care as well as emergency patient care. These hospitals were pioneer health systems in the country to care for COVID-19–positive patients. They created a special ward for COVID-19–infected patients in February 2020. They served as a quarantine site for confirmed cases and established a center to monitor and adapt to rapidly changing conditions and recommendations from local, state, and federal officials. The hospital characteristics are summarized in Table 1. These hospitals have implemented one of the main commercially available EHR systems (brands) in Iran with similar applications, functionalities, and features. Some of these vendors provide services to public hospitals, and some offer services to private hospitals. All EHR systems offer some of the basic features such as viewing laboratory results, collecting patient demographics, listing patient problems, and highlighting out-of-range laboratory results. However, other advanced features such as sending and receiving reminders and messages and remote access to health information have been scarcely or not implemented yet. Table 2 shows the prevalence of EHR features implemented in the hospitals of interest in this study.

Table 1. Hospital details.
Type of hospitalPhysicians, nNurses, nOutpatients in a year, nTotal patients (annually), nBeds, n
Private hospital15018516,00023,00050
Private hospital9017015,00020,00045
Private hospital7018015,00022,00050
Public hospital25040011,00015,000330
Private hospital18032020,00031,000445
Private hospital12040024,00030,000200
Table 2. Implementation of EHR features among the hospitals in this study.
EHRa featureHospitals (N=6), n
View laboratory results electronically6
Collect patient demographics electronically6
List patient problems electronically6
Highlight out-of-range laboratory results6
List patient medications electronically5
Create medical history and follow-up notes5
Order laboratory tests electronically5
Order radiology tests electronically5
View electronic clinical notes electronically4
View imaging results electronically4
View allergy lists4
Warn of drug interactions and contraindications3
Personal human resources (payroll, benefits, training)3
Preregister systems3
Send prescriptions electronically to a pharmacy2
View immunization records1
Send medical information securely to other health care professionals1
Receive reminders for guideline-based interventions1
Receive reminders for preventive screening tests1
Sending electronic referrals1
Receive reminders for screens of chronic disease management0
Sending electronic clinical messaging to patients0
Remote access to EHR0

aEHR: electronic health record.

Research Design

This study was reviewed and approved by the involved hospitals. Data collection was conducted in the six hospitals using an online survey during May and August 2020. For data collection, we could not reach out to all physicians and nurses working in the six hospitals for two reasons. First, due to the COVID-19 pandemic, a large number of clinicians did not work in the same shifts as they worked before the outbreak. Second, many physicians and nurses of the hospitals did not work at the COVID-19 specialized wards. Therefore, our clinician population size was shortened to clinicians working in different shifts at the COVID-19–dedicated wards. Based on the information provided by the six hospitals’ administrations, the total population size was 586 clinicians. We asked the administrators of the COVID-19 specialized wards to distribute the survey online among all clinicians working in different shifts at the COVID-19 wards during May and August 2020 (4 months).

Since WhatsApp is a popular communication app in Iran, the survey was mainly administered via WhatsApp to physicians (with different specialties) and nurses who were directly exposed to patients who were suspected of or infected with COVID-19. The survey was designed to measure facets of HIT use, including EHR functionality, electronic prescribing, health information exchange, other technology-based tools, and informatics applications.

Survey questions also examined the effects of using HIT on workflow, patient care, and clinician burnout during the COVID-19 pandemic. The survey was first prepared in English; then, it was translated to Farsi by one of the authors. The survey was reviewed by a physician working in Iran, and based on the feedback, some modifications were made for clarity purposes. Next, the survey was translated back to English by a university professor in Iran to ensure the correct terms were used and the questions were clear and understandable to participants. Before data collection, survey questions were reviewed, evaluated, and approved by the IT departments and hospital administrations. Finally, the survey was conducted anonymously at the individual clinician level. The survey questions are provided in Multimedia Appendix 1.

In this study, the dependent variable was clinician burnout. We used a single item to measure clinician burnout. This single question was adapted from the physician’s work–life study [21,22]. The measure has six answers: (1) no symptoms of burnout, (2) under stress but do not feel burned out, (3) having one or more symptoms of burnout such as emotional exhaustion, (4) thinking about work frustrations as symptoms of burnout will not go away, (5) feeling completely burned out and seeking help, and (6) completely burned out and getting help now [23]. Following previous studies [18], we dichotomized this measure into “no symptoms of burnout” (≤2 on the 5-point scale) and “one or more symptoms of burnout” (≥3 on the 5-point scale). This single-item measure was validated for clinicians and exhibited a better sensitivity and specificity compared with the Maslach Burnout Inventory survey [24].

Next, we were interested in identifying the factors that influence burnout during the COVID-19 pandemic. Indicators of burnout include awareness of EHR features, EHR system usability, concern about COVID-19, use of technology solutions, use of hospital technology interventions, hospital preparedness, and professional efficacy. Table 3 shows the description of each indicator.

Table 3. Description and source of variables used in this study.
MeasuresDescriptionSources
Awareness of EHRa featuresThe extent to which participants are aware of available EHR features[25]
EHR system usabilityAdapted from the System Usability Scale: giving a global view of subjective assessments of EHR system usability[26]
Concerned about COVID-19The level of concern of the participants about the effect of the COVID-19 pandemic in their life[7]
Use of technology solutionsThe extent to which individuals use technology solutions for coping with work-related stress[27,28]
Use of hospital technology interventionsThe extent to which individuals agree with the use of different organizational interventions at their workplace (hospital)[29,30]
Perceived hospital preparednessThe extent to which clinician perceives the hospital is prepared to deal with the COVID-19 crisis[31]
Professional efficacyAdapted from Maslach Burnout Inventory-General Survey: the extent to which an individual’s perceived effectiveness and accomplishment at work[32]

aEHR: electronic health record.

We included several control variables such as patients’ demographics and practice setting. Respondents provided information about their age, gender, marital status, years of practice, role, specialty, area of work, and hospital setting.

There is considerable debate in the health informatics community about the actual role health information technologies play in clinician burnout [1]. We are interested in identifying the specific factors related to EHRs that may cause stress among clinicians. Further, once we understand the causes of burnout, we are left to ponder innovative solutions to prevent or mitigate burnout. Existing research suggests that some technological advances may help to reduce burnout. However, more studies need to investigate the confounding effect of regulatory mandates and interventions that aim at improving communication, changing workflow, or addressing clinician concerns via quality improvement projects [33]. Therefore, we added additional questions in the survey to understand EHR-related causes of stress, other technology-based solutions, and policy interventions that may help clinicians reduce burnout. Respondents were asked to report stressors associated with the use of EHRs on a 5-point scale. The factors causing stress as a result of using EHRs were increasing computerization of practice, too much time spent on EHRs at work, spending an enormous amount of time on the EHRs at home, insufficient time for documentation, too much data entry, having less face-to-face time for conversation and examination with patients, and inadequate technology-related training for using EHRs at work. These measures were adopted from previous studies [23].

Using a 5-point scale, participants were asked to indicate to what extent using technology-based solutions can help them cope with work-related stress. The solutions were availability of training programs; availability of responsive IT support; possibility for telecommuting (working from home); using technology and tools to follow up with patients remotely (telemedicine); use of mobile apps for meditation, breathing, and relaxation; availability of help desk services; mentorship programs through teleconsultation; and communication groups via messaging apps (eg, WhatsApp, Telegram, and Viber).

Using a 5-point scale, we asked respondents to evaluate the effectiveness of the hospital’s policies, strategies, and interventions regarding the implementation and use of technology-based tools and informatics infrastructure at the workplace. The policies and interventions were developing standards for order entry and reporting among hospitals; using data analysts (specialists) to analyze patient data and elaborate on the patterns; designing an online survey to regularly measure satisfaction with technology and possible technology-related risks and stress; establishing a regular assessment to evaluate the effectiveness of technology in hospitals; using a systematic way to measure workplace burnout and analyze the results; formulating transparent policies, expectations, and goals regarding the use of technology in the clinical workflow; applying clear regulations about the responsibility of clinicians when medical errors occur using technology; and providing incentives for using technology meaningfully for health care delivery. These items were followed by an open box for the participants to add effective interventions or strategies.

Statistical Analysis

We used SPSS Statistics V21.0 (IBM Corp) to conduct all statistical analyses. We used univariable statistics to describe the respondents’ demographic characteristics, the prevalence of burnout, general causes of stress at the workplace, EHR-related stress, and technology solutions to cope with stress. We used multivariable logistic regression to measure the association between burnout and measures of respondents’ awareness of available EHR features, perceived EHR system usability, level of concerns about COVID-19, use of technology solutions for coping with stress, use of technology at the workplace, hospital preparedness for the pandemic, and perceived professional efficacy. We controlled the model for respondents’ age, gender, marital status, role, area of work, specialty, and the number of years they have been in practice to ensure that these factors did not create bias in the results. We performed a sensitivity analysis using an ordered logit model to examine whether the dependent variable (burnout) was represented by its ordinal response categories instead of the dichotomized response categories included in the primary analysis. We also tested the burnout association with the number of patients with COVID-19 that have been cared for by the respondents to identify whether more patients are independently associated with burnout. The results of the sensitivity analysis are reported in Multimedia Appendix 2. Further, to ensure the robustness of the coefficients, we repeated the analysis by considering within-hospital correlations and controlling for the hospital effect. Results are reported in Multimedia Appendix 3 and show that all coefficients remain robust.


Due to a large amount of abrupt tension and irregularity caused by the pandemic, only 373 clinicians initially attempted the survey. After removing incomplete answers, we finally collected 368 fully completed surveys that included 147 nurses, 161 physicians, and 60 physician assistants providing care for patients with COVID-19 (Table 4). The majority of the respondents were female (n=266, 72%), and 68% (n=252) were married. Among the respondents, 22% (n=80) were younger than 35 years, 42% (n=154) were aged between 35 and 44 years, 21% (n=78) were aged between 45 and 54 years, and 16% (n=56) were 55 years or older. There was about an equal number of respondents working in emergency departments, intensive care, and operating rooms (all n=67, 18%), while 34% (n=124) reported working in other inpatient services, and 12% (n=43) reported working in outpatient services. About a quarter of the respondents reported having less than 5 years of experience in practice, while 20% (n=73) reported between 6 to 10 years of experience, 19% (n=71) reported between 11 to 15 years of experience, 22% (n=82) between 16 and 20 years of experience, and 13% (n=46) reported having more than 20 years of experience in practice. The majority of the respondents were from private hospitals (n=310, 84%). About 43% (n=159) of the respondents reported having at least one extra night shift per week since the beginning of the pandemic.

Table 4. Characteristics of respondents.
CharacteristicsSample (N=368), n (%)
Age (years)

<3580 (22)

35-44154 (42)

45-5478 (21)

55-6435 (10)

≥6521 (6)
Gender

Male102 (28)

Female266 (72)
Marital status

Married252 (68)

Single116 (32)
Role

Nurse147 (40)

Physician161 (44)

Physician assistant60 (16)
Area of work

Emergency department67 (18)

Intensive care unit67 (18)

Other inpatient services124 (34)

Outpatient services43 (12)

Operating rooms67 (18)
Specialty

Emergency medicine27 (7)

Family medicine38 (10)

Surgery56 (16)

Anesthesiology67 (18)

Gynecology50 (14)

Nursing56 (14)
Years in practice

<13 (0.01)

1-596 (26)

6-1073 (20)

11-1571 (19)

16-2082 (22)

>2046 (13)
Hospital setting

Public hospital58 (16)

Private hospital310 (84)
Burnout prevalence

I enjoy my work. I have no symptoms of burnout.115 (31)

I am under stress and don’t always have as much energy as I did, but I don’t feel burned out.119 (32)

I am definitely burning out and have one or more symptoms of burnout (eg, emotional exhaustion).66 (18)

The symptoms of burnout I am experiencing won’t go away. I think about work frustrations a lot.24 (7)

I feel completely burned out. I am at the point where I may need to seek help.28 (8)

I am completely burned out, and I am getting help.16 (4)

A total of 36% (134/368) of the respondents reported having at least one symptom of burnout. The highest level of burnout was reported among the physician assistants (32/60, 53%), followed by nurses (Figure 1).

Regarding the area of work, most burnout symptoms were reported in intensive care units, followed by emergency department and other outpatient services (Figure 2).

Figure 1. Percent of respondents reporting at least one symptom of burnout in their role.
View this figure
Figure 2. Percent of respondents reporting at least one symptom of burnout by area of work.
View this figure

The specialties who experienced the most from burnout practiced family medicine and emergency medicine, followed by surgery and anesthesiology (Figure 3). Results show that family medicine physicians experienced the most from burnout. This finding is in line with several previous studies [34]. Family medicine physicians usually feel the most stress because they need to provide comprehensive health care for people of all ages. During the pandemic, family medicine visited more patients with similar symptoms (eg, common cold or flu). At the time of our study, people who felt that they might be at risk of being COVID-19 positive usually visited their trusted family medicine first for a check-up or advice. This also has some cultural reasons because family doctors are generally considered the most trusted physicians for families and the first choice to visit in case of an ailment. Thus, family medicine physicians visited various patients across all ages, genders, and diseases, and they felt the most stress and frustration under the pandemic situation.

Figure 3. Percent of respondents reporting at least one symptom of burnout by specialty.
View this figure

Before running the logistic regression model, we assessed the reliability of the proposed measures. Table 5 presents some indicators and descriptive statistics for each measure. We assessed the reliability using Cronbach alpha for all measures. The recommended threshold value is above .70 [35], which implies adequate reliability for all measures in this study. For awareness of EHR features, since it was measured as a set of dichotomous items, we separately calculated an aggregate score adjusted for each hospital and used that score in the logistic regression model to assess its impact on burnout. Higher scores show a higher level of awareness about the available EHR features.

Table 5. Summary of the properties of the measurements.
MeasuresMeasurement typeMean (SD)Cronbach α
EHRa system usability10-item scale measured on a 5-point Likert scale3.20 (0.71).85
Concerned about COVID-194 items measured on 5-point Likert scale4.43 (0.67).72
Use of technology solutions8 items measured on a 5-point Likert scale3.12 (0.80).81
Use of hospital technology interventions8 items measured on a 5-point Likert scale2.89 (0.85).86
Hospital preparedness5 items measured on a 5-point Likert scale3.50 (0.86).87
Professional efficacy4 items measured on a 5-point Likert scale3.75 (0.79).88

aEHR: electronic health record.

Next, based on the multivariable logistic regression model while controlling for age, gender, marital status, role, area of work, specialty, years of experience, and type of hospital (Table 6), we identified that perception of EHR system usability, level of concern about COVID-19, hospital technology-related interventions, and level of hospital preparedness to cope with the pandemic are the significant factors associated with burnout among the clinicians. Positive perceptions about the ease of use of EHRs and a high level of confidence in understanding its function were associated with lower odds of burnout symptoms (OR –0.16, 95% CI –0.24 to –0.09; P<.001). In other words, the nurses and physicians who find EHRs useable and are confident in their knowledge and ability to work with it have lower odds of having any burnout symptoms compared to those who find EHRs difficult and cumbersome to use. Similarly, the more use of technology interventions implemented in the hospitals was associated with lower odds of burnout (OR –0.38, 95% CI –0.51 to –0.26; P<.001).

Table 6. The estimate of the association between demographic, practice, and EHR characteristics and 1 or more symptoms of burnout.
VariableORa (95% CI)SEZ valueP value
Awareness of EHRb features0.03 (–0.07 to 0.13)0.050.54.59
EHR system usability–0.16 (–0.24 to –0.09)0.04–4.16<.001
Concerned about COVID-190.29 (0.11 to 0.48)0.093.05.002
Use of technology solutions–0.06 (–0.14 to 0.03)0.04–1.31.19
Use of hospital technology interventions–0.38 (–0.51 to –0.26)0.06–5.95<.001
Hospital preparedness–0.23 (–0.38 to –0.09)0.07–3.05.002
Professional efficacy–0.06 (–0.23 to 0.11)0.09–0.70.49
Age (years; reference group: ≥65 years)

<35–0.11 (–3.48 to 3.09)1.65–0.07.95

35-44–2.13 (–5.49 to 0.79)1.58–1.35.18

45-54–0.88 (–4.05 to 1.98)1.51–0.59.56

55-641.61 (–1.35 to 4.73)1.511.06.29
Gender (reference group: female)

Male–0.75 (–1.98 to 0.44)0.61–1.22.22
Marital status (reference group: single)

Married–0.30 (–1.35 to 0.72)0.53–0.57.57
Role (reference group: physician assistant)

Nurse–0.93 (–2.24 to 0.34)0.66–1.42.16

Physician–2.34 (–3.96 to –0.86)0.78–2.99.003
Area of work (reference group: operating rooms)

Emergency department–1.38 (–2.89 to 0.05)0.75–1.85.07

Intensive care unit0.55 (–1.06 to 2.2)0.830.66.51

Other inpatient services–1.96 (–3.27 to –0.73)0.65–3.04.002

Outpatient services–1.18 (–2.95 to 0.49)0.87–1.36.18
Specialty (reference group: nursing)

Emergency medicine–1.47 (–3.61 to 0.49)1.04–1.41.16

Family medicine–0.76 (–2.8 to 1.27)1.03–0.73.46

Surgery–2.25 (–4.42 to –0.25)1.05–2.14.03

Anesthesiology–1.14 (–2.41 to 0.09)0.63–1.80.07

Gynecology0.83 (–0.62 to 2.34)0.751.11.27
Years in practice (reference group: 16-20)

<1–1.25 (–3.57 to 1.12)1.18–1.06.29

1-51.33 (–0.79 to 3.61)1.111.21.23

6-10–0.93 (–3.02 to 1.2)1.06–0.88.38

11-15–0.12 (–2.12 to 1.95)1.02–0.12.91
Type of hospital (reference group: public)

Private–0.21 (–1.65 to 1.19)0.71–0.29.77

(Intercept)11.30 (6.35 to 16.86)2.654.27<.001

aOR: odds ratio.

bEHR: electronic health record.

We realize that the nurses and physicians working in hospitals that implemented various interventions regarding the use of technology at the workplace reveal lower odds of having any symptom of burnout compared to those working in hospitals that did not implement such interventions. Hospital preparedness was also associated with a reduction in burnout symptoms (OR –0.23, 95% CI –0.38 to –0.09; P=.002). Regarding the level of concern about COVID-19, we identified that the more the clinicians are concerned about the pandemic situation, the higher the odds of burnout (OR 0.29, 95% CI 0.11-0.48; P=.002). We did not find a significant association between the odds of having at least one symptom of burnout and the awareness about EHR features, use of technology solutions, and respondents’ professional efficacy.

Among the control variables, physicians (as a role), other inpatient services (as an area of work), and surgery (as a specialty) exhibited significantly lower burnout symptoms. Among the work areas, health care workers in other inpatient centers (eg, services delivered in rehabilitation centers, psychiatric departments, or addiction treatment centers) exhibited lower burnout.

We asked respondents about the factors that cause stress in their work in general prior to the pandemic. Almost one-fifth of the participants listed “insufficient compensation” or “family responsibilities” as the leading cause of stress. Almost one-fourth of the participants listed “workload” and almost one-fifth listed “health risks” as the second most important stressors. During the pandemic, the clinicians’ workload increased, and the concerns over contracting the virus or transmitting it to family members amplified. Almost one-fifth of the participants reported that they have work on average about 20 hours per week during the pandemic compared to the before the pandemic. Almost one-fourth of the participants reported that they have to work over the weekends more often during the pandemic compared to the time before it. Almost half of the participants reported an increase in the number of nights that they needed to be on call per week during the pandemic.

We were also interested in understanding the EHR-related causes of stress during the pandemic. Figure 4 shows several EHR-related stress factors during the pandemic. During the pandemic, an extension was made to EHRs to register and keep track of COVID-19 tests administered and to send the results electronically to a national online repository. For the tests with positive results, the record of hospitalization or death was populated in the system. Respondents strongly agreed that the most important leading cause of EHR-related stress is inadequate training for using technology, followed by having less face-to-face time with patients, too much time spent on data entry, and increasing computerization at work. Since during the pandemic clinicians’ shifts changed and they had an additional workload with an unclear use policy, the likelihood of creating errors through using EHRs increased. Since the implementation of EHRs is still at an early stage, respondents mentioned inadequate training for using EHR features. This training issue, coupled with other key issues such as having less face-to-face time with patients (due to social distancing) and involving too much data entry (COVID-19 test results), caused more stress during the pandemic compared to normal times since the workload increased and job pressure was higher. The situational factors (due to the pandemic) exacerbated stress associated with EHR use in hospitals.

Figure 4. Leading causes of EHR-related stress during the COVID-19 pandemic. EHR: electronic health record.
View this figure

Next, we asked participants about the effectiveness of implementing different interventions at the hospitals during the pandemic. Respondents believed that the hospitals’ most effective intervention was implementing clear regulations about the responsibility of clinicians when medical errors occur using technology. Clinicians are more prone to medical errors when they work under pressure and stress imposed by the pandemic. Therefore, the hospitals need to develop and implement proper regulations and guidelines, and communicate them clearly with the clinicians. Other effective interventions include imposing transparent policies, expectations, and goals regarding the use of technology in the clinical workflow and developing unified standards for order entry and reporting among hospitals (Figure 5).

Figure 5. Effective hospital interventions during the COVID-19 pandemic.
View this figure

The most effective solutions that the respondents selected for addressing their burnout symptoms were using communication tools that facilitate group interactions (eg, WhatsApp and Telegram) and providing training programs for learning EHR systems. Respondents identified the second most effective solution as the availability of telecommuting (working from home), followed by participating in mentorship programs through teleconsultation (Figure 6). These findings show that the most prominent solution for reducing burnout among clinicians during the pandemic is facilitating the communication and information channels for them in a fast-paced environment to get the timely information they need to perform their activities. To practice social distancing, which is one of the main factors that can help reduce the transmission of COVID-19, the clinician needs to provide some of the care services remotely through telemedicine or teleconsultation. Our results show that special training to facilitate these activities for the clinician seems to be among the essential solutions to reduce clinicians’ burnout.

Figure 6. Effective technology solutions during the COVID-19 pandemic.
View this figure

Statement of Principal Findings

The COVID-19 pandemic imposed serious challenges to the health care systems of numerous countries. As this virus spreads worldwide, more research is needed to address clinicians’ burnout during the pandemic. This study aims to investigate burnout issues among physicians and nurses working in Iranian hospitals that directly provide care for patients suspected of or infected with COVID-19. In this study of 368 clinicians working in hospitals in Tehran, we found that EHR usability, technology-based hospital interventions, hospital preparedness, and level of concern about COVID-19 were significantly associated with workplace burnout. This finding is consistent with previous studies suggesting poor EHR usability will lead to clinicians’ dissatisfaction and, in turn, burnout [36]. EHRs have also been found to be a useful tool to support outbreak management [16]. Clinicians who believe that EHRs are not unnecessarily complicated and that they can have the support of technical personnel to use the system are less likely to experience burnout symptoms. Moreover, clinicians who consider that the various functions in EHRs are well integrated and that they can learn the features quickly are less prone to exhibit burnout symptoms. Given the negative association between EHR usability and burnout, hospitals need to improve EHR user-friendliness and convenience to reduce clinician burnout. Based on this study’s findings, we recommend that hospitals measure EHR usability and burnout among their clinicians in a regular and systematic way. Those clinicians who reported higher burnout symptoms could be the subject of a focus group to determine what changes are required by hospitals in implementing EHRs (for instance, availability of EHR-related training or technical support).

Interpretation Within the Context of the Wider Literature

This study shows that the more caregivers are familiar with the EHR systems and feel confident in their ability to use them, the less they feel burnout symptoms. Interestingly in our sample, the most effective solution for reducing burnout symptoms was providing a training program for EHR use. Our results are in line with recent studies on EHR-related perceptions of workload that suggest individualized EHR training can improve the knowledge of EHR tools and satisfaction [37,38]. Other studies indicated that a considerable amount of time that the care providers need to spend recording information in the EHR system and data entry was the essential contributor to burnout [18]. However, our study reveals that the respondents are less worried about the data entry and their time using the system either at work or at home. They feel that more useful education and training programs are required to improve their symptoms of burnout. We also call attention to other important solutions to address burnout, such as easily accessible mentorship programs through teleconsultation and the availability of responsive IT support. Advanced EHR education through highly interactive personalized mentorship and hands-on workshops can help care providers alleviate workplace burnout and improve care quality [39,40]. EHR mentorship programs may improve the care providers’ self-efficacy, and the availability of 24-hour IT support provides the peace of mind that help is always accessible for them. This solution will help them combat burnout symptoms and psychological stress.

Our findings suggest that technology-related hospital interventions can decrease burnout among clinicians. We highlight several potential interventions to reduce the odds of burnout. Hospitals need to develop shared standards for data entry and exchanging health information among hospitals [41]. This intervention plays a significant role in the pandemic since prompt information sharing between hospitals and other health care organizations is required. Hospitals can also recruit business intelligence [38] specialists to analyze patient data, extract patterns, and illuminate risk factors and trends. Using reporting and analytics tools during the outbreak, data analysts can help hospitals devise better strategies by analyzing patient data from different perspectives (eg, location, symptoms, and contact with COVID-19–positive patients). Another possible intervention by hospitals could be using an online survey to regularly measure satisfaction with the technology used in their clinical setting. This approach can help hospitals identify potential technology-related risks and stress, reducing the likelihood of clinician burnout. Furthermore, hospitals can frequently evaluate the effectiveness of current technology. This approach provides hospitals with a better assessment of technology from the users’ perspectives. For instance, during the pandemic, hospitals may recognize the need to implement multiple COVID-19–specific tools (eg, an EHR-integrated patient self-triage and self-scheduling tool to manage COVID-19) [8].

Implications for Policy, Practice, and Research

Based on the findings, a significant hospital intervention could be applying a systematic method for measuring workplace burnout and analyzing the results. Clinician burnout surveys could be conducted frequently. However, the findings and feedback could remain uninvestigated, and it will be unclear how burnout affects health care professionals’ productivity [42]. For instance, during the COVID-19 pandemic, clinicians may encounter new stressors that might not have been recognized previously [7]. Discovering stressors and analyzing the source of stress (eg, technology-related) can be useful for a rapid response to clinicians’ burnout during the COVID-19 outbreak.

Hospitals should establish transparent policies, expectations, and goals regarding the use of technology-based tools in the clinical workflow. For instance, the primary purpose of EHR implementation, the extent to which EHR features should be used, and clear regulations and guidelines for using EHRs should be communicated to clinicians working in hospitals. Otherwise, the lack of clear use strategies regarding clinician–technology interactions can lead to confusion, stress, and burnout [43]. First, we recommend that hospitals clarify clinicians’ responsibility, accountability, and associated regulations when medical errors occur using technology (eg, EHRs) in the clinical setting. Second, hospitals should define an EHR meaningful use program to elucidate what features of EHRs are essential and why their use is mandatory and, moreover, what applications are advanced or voluntary. Third, policy makers can offer incentives to hospitals and clinicians for the meaningful use of technology-based tools for health care delivery. These guidelines and instructions not only set a direction for hospitals on how to successfully implement and use EHRs but also mitigate the likelihood of clinician burnout. We need to acknowledge that any new policies, instructions, and procedures could be sources of burnout at the beginning (during a pandemic in particular). The success factor is implementing the new strategies, guidelines, and processes based on a robust execution plan, enough resources, and transparent communications and collaborations among clinicians in hospitals.

Clinicians working in the COVID-19 ward are significantly worried about becoming infected or concerned about their family becoming infected. This high level of concern about the pandemic significantly raises their burnout symptoms. The findings also demonstrate that hospital preparedness can enable the efficiencies of the COVID-19 care management program by reducing clinician burnout symptoms. Burnout is a system problem [44], and hospitals need to pose rapid solutions to address it during the pandemic. Hospitals can mitigate the odds of burnout among physicians and nurses by providing useful pandemic training, protective equipment, and support programs. Pandemic training (eg, classes, brochures, and meetings) can affect clinician stress by disseminating helpful statistics, insights, and information about the epidemic [45]. Through protective equipment (eg, personal protective equipment, masks, gloves, and sanitizer), hospitals are able to decrease workforce anxiety by providing clinicians facing patients with COVID-19 with various tools to avoid infection [46]. Support programs (eg, online safety alerts and wellness guidance, digital access to benefits, and work from home practices) can reduce burnout by improving work flexibility and helping clinicians access EHRs and other technology across distances [47].

Limitations

This study is subject to some limitations. First, due to data collection limitations, our research is conducted based on a small group of physicians and nurses facing COVID-19. Second, this study is conducted in a resource-limited country with different pandemic preparedness plans and technology infrastructure in the health care system. Therefore, caution should be exercised when generalizing the findings to other countries. Third, because this study was conducted in the critical episode of the epidemic in Iran, selection bias may have skewed our results if highly stressed clinicians or those with little stress decided not to participate. Fourth, we conducted data collection electronically; thus, clinicians who were more comfortable with computers or mobile devices were more likely to participate. Finally, due to the time of data collection, our findings may not be generalized to the onset or the time that COVID-19 is under control in Iran. We recommend that future studies extend our research in the same or other contexts by collecting more comprehensive data using different data collection strategies, considering additional informatics-based tools, and examining further epidemic factors at different times.

Conclusions

Studying technology-related factors affecting clinicians’ burnout has attracted much attention in the health care setting. This study adds to the literature by examining how EHR usability, technology-based hospital interventions, hospital preparedness, and concern about COVID-19 are significantly associated with burnout among clinicians during the COVID-19 pandemic in a resource-limited country. The results show the critical role of transparent hospital policies, EHR implementation strategies, training needs, and hospital pandemic support programs in reducing burnout. Interestingly, transparency is identified as the most important intervention that hospitals should implement to address burnout symptoms. Transparency in regulations addressing medical errors using technology and transparency dealing with policies and expectations of clinical workflows were identified as critical factors to alleviate workplace burnout. Due to the critical effects of burnout on physicians, nurses, patients, and the health care system, hospitals need to design robust interventions to address stress generated by HIT. Through this process, hospitals can mitigate the clinician burden and improve quality of care and patient safety.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Survey questions.

DOCX File , 27 KB

Multimedia Appendix 2

Sensitivity analysis.

DOCX File , 19 KB

Multimedia Appendix 3

Results after controlling for a hospital effect.

DOCX File , 23 KB

References

  1. National Academies of Sciences, Engineering, and Medicine. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. Washington, DC: The National Academies Press; 2019.
  2. Salyers MP, Bonfils KA, Luther L, Firmin RL, White DA, Adams EL, et al. The relationship between professional burnout and quality and safety in healthcare: a meta-analysis. J Gen Intern Med 2017 May;32(4):475-482 [FREE Full text] [CrossRef] [Medline]
  3. Friedberg M, Chen P, Van Busum KR, Aunon F, Pham C, Caloyeras J, et al. Factors affecting physician professional satisfaction and their implications for patient care, health systems, and health policy. Rand Health Q 2014;3(4):1 [FREE Full text] [Medline]
  4. Shanafelt T, Noseworthy JH. Executive leadership and physician well-being: nine organizational strategies to promote engagement and reduce burnout. Mayo Clin Proc 2017 Jan;92(1):129-146. [CrossRef] [Medline]
  5. He F, Deng Y, Li W. Coronavirus disease 2019: what we know? J Med Virol 2020 Jul;92(7):719-725 [FREE Full text] [CrossRef] [Medline]
  6. Adams JG, Walls RM. Supporting the health care workforce during the COVID-19 global epidemic. JAMA 2020 May 21;323(15):1439-1440. [CrossRef] [Medline]
  7. Wu Y, Wang J, Luo C, Hu S, Lin X, Anderson AE, et al. A comparison of burnout frequency among oncology physicians and nurses working on the frontline and usual wards during the COVID-19 epidemic in Wuhan, China. J Pain Symptom Manage 2020 Jul;60(1):e60-e65 [FREE Full text] [CrossRef] [Medline]
  8. Judson T, Odisho A, Neinstein A, Chao J, Williams A, Miller C, et al. Rapid design and implementation of an integrated patient self-triage and self-scheduling tool for COVID-19. J Am Med Inform Assoc 2020 Jun 01;27(6):860-866 [FREE Full text] [CrossRef] [Medline]
  9. Chen Q, Liang M, Li Y, Guo J, Fei D, Wang L, et al. Mental health care for medical staff in China during the COVID-19 outbreak. Lancet Psychiatry 2020 Apr;7(4):e15-e16 [FREE Full text] [CrossRef] [Medline]
  10. Koh D. Occupational risks for COVID-19 infection. Occup Med (Lond) 2020 Mar 12;70(1):3-5 [FREE Full text] [CrossRef] [Medline]
  11. Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform 2014 Dec;83(11):779-796. [CrossRef] [Medline]
  12. Triantafillou P. Making electronic health records support quality management: a narrative review. Int J Med Inform 2017 Aug;104:105-119. [CrossRef] [Medline]
  13. Guan W, Ni Z, Hu Y, Liang W, Ou C, He J, et al. Clinical characteristics of coronavirus disease 2019 in China. N Engl J Med 2020 Apr 30;382(18):1708-1720. [CrossRef]
  14. Amir-Behghadami M, Janati A. Iranian national COVID-19 electronic screening system: experience to share. Emerg Med J 2020 Jul;37(7):412-413 [FREE Full text] [CrossRef] [Medline]
  15. Lenert L, McSwain BY. Balancing health privacy, health information exchange, and research in the context of the COVID-19 pandemic. J Am Med Inform Assoc 2020 Jun 01;27(6):963-966 [FREE Full text] [CrossRef] [Medline]
  16. Reeves J, Hollandsworth H, Torriani F, Taplitz R, Abeles S, Tai-Seale M, et al. Rapid response to COVID-19: health informatics support for outbreak management in an academic health system. J Am Med Inform Assoc 2020 Jun 01;27(6):853-859 [FREE Full text] [CrossRef] [Medline]
  17. Bakken S. Can informatics innovation help mitigate clinician burnout? J Am Med Inform Assoc 2019 Feb 01;26(2):93-94 [FREE Full text] [CrossRef] [Medline]
  18. Gardner R, Cooper E, Haskell J, Harris D, Poplau S, Kroth P, et al. Physician stress and burnout: the impact of health information technology. J Am Med Inform Assoc 2019 Feb 01;26(2):106-114 [FREE Full text] [CrossRef] [Medline]
  19. Johns Hopkins Coronavirus Resource Center.   URL: https://coronavirus.jhu.edu/ [accessed 2021-04-08]
  20. Faiz S, Riahi T, Rahimzadeh P, Nikoubakht N. Commentary: remote electronic consultation for COVID-19 patients in teaching hospitals in Tehran, Iran. Med J Islam Repub Iran 2020;34:31 [FREE Full text] [CrossRef] [Medline]
  21. Schmoldt RA, Freeborn DK, Klevit HD. Physician burnout: recommendations for HMO managers. HMO Pract 1994 Jul;8(2):58-63. [Medline]
  22. Williams ES, Konrad TR, Linzer M, McMurray J, Pathman DE, Gerrity M, et al. Refining the measurement of physician job satisfaction: results from the Physician Worklife Survey. SGIM Career Satisfaction Study Group. Society of General Internal Medicine. Med Care 1999 Dec;37(11):1140-1154. [CrossRef] [Medline]
  23. Britt HR, Koranne R, Rockwood T. Statewide improvement approach to clinician burnout: findings from the baseline year. Burnout Res 2017 Dec;7:29-35. [CrossRef]
  24. Dolan ED, Mohr D, Lempa M, Joos S, Fihn SD, Nelson KM, et al. Using a single item to measure burnout in primary care staff: a psychometric evaluation. J Gen Intern Med 2015 May;30(5):582-587 [FREE Full text] [CrossRef] [Medline]
  25. Häyrinen K, Saranto K, Nykänen P. Definition, structure, content, use and impacts of electronic health records: a review of the research literature. Int J Med Inform 2008 May;77(5):291-304. [CrossRef] [Medline]
  26. Brooke J. SUS: a 'quick and dirty' usability scale. In: Jordan PW, Thomas B, McClelland IL, Weerdmeester B, editors. Usability Evaluation In Industry. Boca Raton, FL: CRC Press; 1996:189.
  27. Brennan F. Technostress and leadership: a case study in higher education during the COVID-19 crisis. Theseus. 2021.   URL: https://www.theseus.fi/bitstream/handle/10024/380031/Brennan_Fintan%20.pdf?sequence=3&isAllowed=y [accessed 2021-04-12]
  28. Vaziri H, Casper WJ, Wayne JH, Matthews RA. Changes to the work-family interface during the COVID-19 pandemic: examining predictors and implications using latent transition analysis. J Appl Psychol 2020 Oct;105(10):1073-1087. [CrossRef] [Medline]
  29. Christen P, D'Aeth JC, Løchen A, McCabe R, Rizmie D, Schmit N, et al. The J-IDEA Pandemic Planner: a framework for implementing hospital provision interventions during the COVID-19 pandemic. Med Care 2021 May 01;59(5):371-378 [FREE Full text] [CrossRef] [Medline]
  30. Mahmud E, Dauerman HL, Welt FG, Messenger JC, Rao SV, Grines C, et al. Management of acute myocardial infarction during the COVID-19 pandemic: a position statement from the Society for Cardiovascular Angiography and Interventions (SCAI), the American College of Cardiology (ACC), and the American College of Emergency Physicians (ACEP). J Am Coll Cardiol 2020 Sep 15;76(11):1375-1384 [FREE Full text] [CrossRef] [Medline]
  31. Responding to community spread of COVID-19: interim guidance, 7 March 2020. World Health Organization.   URL: https://apps.who.int/iris/handle/10665/331421 [accessed 2021-02-10]
  32. Schaufeli W, Leiter M, Maslach C, Jackson S. Maslach Burnout Inventory-General Survey. In: Maslach C, Jackson SE, Leiter MP, editors. The Maslach Burnout Inventory: Test Manual. Palo Alto, CA: Consulting Psychologists Press; 1996.
  33. Physician burnout. Agency for Healthcare Research and Quality.   URL: https://www.ahrq.gov/prevention/clinician/ahrq-works/burnout/index.html [accessed 2021-04-02]
  34. Shanafelt T, Hasan O, Dyrbye L, Sinsky C, Satele D, Sloan J, et al. Changes in burnout and satisfaction with work-life balance in physicians and the general US working population between 2011 and 2014. Mayo Clin Proc 2015 Dec;90(12):1600-1613. [CrossRef] [Medline]
  35. Nunnally JC. An overview of psychological measurement. In: Wolman BB, editor. Clinical Diagnosis of Mental Disorders: A Handbook. Boston, MA: Springer; 1978.
  36. Babbott S, Manwell LB, Brown R, Montague E, Williams E, Schwartz M, et al. Electronic medical records and physician stress in primary care: results from the MEMO Study. J Am Med Inform Assoc 2014 Mar;21(e1):e100-e106 [FREE Full text] [CrossRef] [Medline]
  37. DiAngi YT, Stevens LA, Halpern-Felsher B, Pageler NM, Lee TC. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA Open 2019 Jul;2(2):222-230 [FREE Full text] [CrossRef] [Medline]
  38. Robinson K, Kersey JA. Novel electronic health record (EHR) education intervention in large healthcare organization improves quality, efficiency, time, and impact on burnout. Medicine (Baltimore) 2018 Oct;97(38):e12319. [CrossRef] [Medline]
  39. Jordan J, Watcha D, Cassella C, Kaji AH, Trivedi S. Impact of a mentorship program on medical student burnout. AEM Educ Train 2019 Jul;3(3):218-225 [FREE Full text] [CrossRef] [Medline]
  40. Perumalswami CR, Takenoshita S, Tanabe A, Kanda R, Hiraike H, Okinaga H, et al. Workplace resources, mentorship, and burnout in early career physician-scientists: a cross sectional study in Japan. BMC Med Educ 2020 Jul 03;20(1):178 [FREE Full text] [CrossRef] [Medline]
  41. Kaelber DC, Bates DW. Health information exchange and patient safety. J Biomed Inform 2007 Dec;40(6 Suppl):S40-S45 [FREE Full text] [CrossRef] [Medline]
  42. Dewa CS, Loong D, Bonato S, Thanh NX, Jacobs P. How does burnout affect physician productivity? A systematic literature review. BMC Health Serv Res 2014 Jul 28;14:325 [FREE Full text] [CrossRef] [Medline]
  43. Rothenberger DA. Physician burnout and well-being: a systematic review and framework for action. Dis Colon Rectum 2017 Jul;60(6):567-576. [CrossRef] [Medline]
  44. Yates SW. Physician stress and burnout. Am J Med 2020 Feb;133(2):160-164 [FREE Full text] [CrossRef] [Medline]
  45. Ueda M, Martins R, Hendrie P, McDonnell T, Crews J, Wong T, et al. Managing cancer care during the COVID-19 pandemic: agility and collaboration toward a common goal. J Natl Compr Canc Netw 2020 Mar 20:1-4. [CrossRef] [Medline]
  46. Livingston E, Desai A, Berkwits M. Sourcing personal protective equipment during the COVID-19 pandemic. JAMA 2020 May 19;323(19):1912-1914. [CrossRef] [Medline]
  47. Hollander JE, Carr BG. Virtually perfect? Telemedicine for Covid-19. N Engl J Med 2020 May 30;382(18):1679-1681. [CrossRef] [Medline]


EHR: electronic health record
HIT: health information technology
IT: information technology


Edited by G Eysenbach; submitted 04.03.21; peer-reviewed by YC Kato-Lin, KM Kuo; comments to author 24.03.21; revised version received 12.04.21; accepted 28.04.21; published 03.06.21

Copyright

©Pouyan Esmaeilzadeh, Tala Mirzaei. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 03.06.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.