This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.
Telemedicine as a mode of health care work has grown dramatically during the COVID-19 pandemic; the impact of this transition on clinicians’ after-hours electronic health record (EHR)–based clinical and administrative work is unclear.
This study assesses the impact of the transition to telemedicine during the COVID-19 pandemic on physicians’ EHR-based after-hours workload (ie, “work outside work”) at a large academic medical center in New York City.
We conducted an EHR-based retrospective cohort study of ambulatory care physicians providing telemedicine services before the pandemic, during the acute pandemic, and after the acute pandemic, relating EHR-based after-hours work to telemedicine intensity (ie, percentage of care provided via telemedicine) and clinical load (ie, patient load per provider).
A total of 2129 physicians were included in this study. During the acute pandemic, the volume of care provided via telemedicine significantly increased for all physicians, whereas patient volume decreased. When normalized by clinical load (ie, average appointments per day by average clinical days per week), telemedicine intensity was positively associated with work outside work across time periods. This association was strongest after the acute pandemic.
Taking physicians’ clinical load into account, physicians who devoted a higher proportion of their clinical time to telemedicine throughout various stages of the pandemic engaged in higher levels of EHR-based after-hours work compared to those who used telemedicine less intensively. This suggests that telemedicine, as currently delivered, may be less efficient than in-person–based care and may increase the after-hours work burden of physicians.
The COVID-19 pandemic precipitated the rise of telemedicine—defined as the synchronous provision of health care services via telecommunications, either video or audio, to patients at remote sites—as a powerful disrupter of health care delivery [
Prior to the pandemic, studies of the provision of clinical care through the medium of telemedicine identified potential benefits such as improved access to care in underserved regions or communities, better coordination of care, greater convenience, and lower costs [
In this paper we focus on ambulatory physicians’ WOW during a time of rapid telework transition spurred by the COVID-19 pandemic. Our goal is to evaluate the impact of telemedicine practice on ambulatory physicians’ EHR-based WOW during the large-scale rollout of telemedicine in an urban academic hospital system during the COVID-19 pandemic.
New York University Langone Health (NYULH) is a large academic health care system in New York City, with over 8000 health care providers across 4 hospitals and over 500 ambulatory faculty group practices. The system is connected via a single EHR system, Epic, with over 7.5 million active patient accounts. Prior to the COVID-19 pandemic, NYULH offered limited telemedicine services only through pilot programs such as “virtual urgent care” (in emergency medicine), postoperative wound checks (in orthopedics), and some mental health services. Telemedicine for primary care and other routine health services was not available. During the pandemic, NYULH rapidly scaled its telemedicine offerings to include primary care, ambulatory specialty practice, and urgent care. NYULH “virtual health” was comprised of a single, enterprise-wide instance of synchronous, video-based telecommunications encounters between physicians and patients in remote locations accessed through a standardized EHR-based patient portal system and a third-party videoconferencing vendor. This platform provided a unified patient and provider experience between clinical practice sites and across specialties. At the height of the pandemic, this system saw an 8595% increase in monthly telemedicine visits between February (n=1699) and April (n=147,736), with over 2000 unique physicians engaging in video visits [
This is an EHR-based retrospective cohort study including all ambulatory care physicians continuously practicing (defined as at least 5 appointments scheduled per week in the reporting period) at any New York-based NYULH faculty group practice site between January 1, 2020, and August 31, 2020. Nonphysician practitioners (eg, advanced-practice providers) and residents were not included in the study cohort, as with few exceptions, they did not provide telemedicine-based care during this period.
This study was deemed part of a quality improvement and met the criteria for exemption from institutional review board’s review according to NYULH institutional policy. All data were collected as part of routine clinical care and administrative management.
Definitions of key variables associated with study measures and analysis are provided in
Epic metric key terms and variables associated with study measures.
Epic metric | Description | Calculation |
Reporting period | For a month, it starts on the Sunday on or immediately before the 1st and ends on the last Saturday of the month. | = (End date-start date) |
Days with appointments | Percentage of days with at least one appointment within the reporting period. | For a reporting period: |
Appointments per day | Average minutes a provider spent in the system outside of scheduled hours. | For a reporting period: |
Time spent outside scheduled hours | Average minutes a provider spent in the system outside of scheduled hours. | For a reporting period: |
Time spent on unscheduled days | Average minutes a provider spent in the system on days with no scheduled patients. | For a reporting period: |
Derived metric key terms and variables associated with study measures.
Derived metric | Calculation |
Scheduled days | For a reporting period: |
Unscheduled days | For a reporting period: |
Time outside scheduled hours per month | For a reporting period: |
Time on unscheduled days per month | For a reporting period: |
Clinical load | For a reporting period: |
“Work outside work” measure | For a reporting period: |
To evaluate whether the effects of telemedicine intensity were influenced by the evolving stages of the COVID-19 pandemic, we aggregated monthly physician data into the following 3 successive time periods: (1) the prepandemic period of January 1-February 29, 2020; (2) the acute pandemic period of March 1-May 31 (with March 15th representing the date when most NYULH ambulatory practices were closed for in-person visits); and (3) after the acute pandemic period of June 1-August 31, representing the gradual resumption of in-person care.
To create a measure of the relative volume of clinical care physicians provided via telemedicine, we calculated the proportion of total visits per month that were telemedicine-based for each physician (number of video visits per month divided by the total number of all patient visits per month per provider) with values that could range from 0 to 1.
Prior research has found clinical load to be an important predictor of WOW burden [
Derived from EHR user activity logs from Epic, WOW was calculated by adding time outside scheduled hours (ie, the average minutes per day spent in the system outside of scheduled hours on scheduled days, where scheduled hours are determined using Epic Cadence scheduling data plus two 30-minute “buffer” periods added before the start of first appointment and after the end of last appointment) and time on unscheduled days (ie, the average number of minutes per day spent in the system on days with no scheduled patients). WOW was normalized for physicians’ patient load by dividing WOW by clinical load to create a measure reflecting WOW per appointment.
An alternative measure of WOW uses the Epic EHR’s own variable-generated data—PT. PT represents the average number of minutes per day spent in charting activities on weekdays outside a standard (local) 7 AM to 5:30 PM workday and any time on weekends. PT does not include time spent personalizing EHR tools (eg, documentation templates or preferences lists) or time using reporting tools such as SlicerDicer and Reporting Workbench during unscheduled days. Although PT can be used as a marker of after-hours clinical work, recent studies have called into question its accuracy and usefulness for this purpose [
We first computed telemedicine intensity, clinical load, WOW, and WOW per appointment for all physicians in the EHR that met our inclusion criteria. To evaluate whether WOW significantly varied across time periods, we ran one-way ANOVAs on both WOW and WOW per appointment. To evaluate the effect of telemedicine intensity and time period on after-hours work burden, as well as whether the relationship between telemedicine intensity and after-hours work varied across time periods, we conducted a hierarchical linear regression analysis in which the dependent variable was WOW per appointment. We first entered the main effects of telemedicine intensity and pandemic time period, followed by the interaction of telemedicine intensity and pandemic time period. To understand the nature of the interaction of telemedicine intensity and pandemic time period, we partitioned the data by time period and regressed WOW per appointment on telemedicine intensity in each time period. All analyses were conducted using SPSS (version 28; IBM Corp).
We analyzed data on 2129 physicians from January to August 2020. The majority of physicians were from internal medicine subspecialties (eg, cardiology, pulmonology, and geriatrics), followed by ambulatory surgery (including general surgery and surgical subspecialists) and general medicine practice (eg, internal medicine and family medicine;
One-way ANOVAs evaluating whether the average WOW per day and WOW per appointment varied by pandemic time period were significant across physicians (average WOW per day:
Across time periods (before the pandemic, during acute pandemic, and after acute pandemic) telemedicine intensity was positively associated with WOW per appointment (step 1 in
Specialty of included study physicians (N=2129).
Clinical specialty | Values, n (%) |
Internal medicine subspecialty | 671 (31.5) |
Surgery | 377 (17.7) |
General practice (eg, internal medicine and family doctors) | 326 (15.3) |
Pediatrics | 175 (8.2) |
Neurology | 141 (6.6) |
Obstetrician and gynecologist | 134 (6.3) |
Other | 91 (4.3) |
Psychiatry | 72 (3.4) |
Emergency medicine | 68 (3.2) |
Dermatology | 36 (1.7) |
Rehab | 32 (1.5) |
Pain medicine | 6 (0.3) |
Work outside work (WOW) per day and per appointment, by time period.
WOW | Time period | ||||||||||
|
Before pandemic | During acute pandemic | After acute pandemic | ||||||||
|
Median | Mean | 95% CI | Median | Mean | 95% CI | Median | Mean | 95% CI | ||
WOW per day | 27.19 | 34.50 | 33.52-35.47 | 23.96 | 30.20 | 29.50-30.91 | 26.94 | 34.11 | 33.31-34.91 | ||
WOW per appointment | 5.73 | 9.29 | 8.92-9.65 | 7.52 | 11.68 | 11.31-12.05 | 6.04 | 10.03 | 9.70-10.37 |
Hierarchical regression of work outside work (WOW) per appointment.
Study variables | Normalized WOW | ||||||
|
Step 1 | Step 2 | |||||
|
Unstandardized coefficient | Standard error | Unstandardized coefficient | Standard error | |||
COVID-19 time period | –0.27 | 0.13 | 0.05 | –0.52 | 0.15 | <.001 | |
Telemedicine intensity | 6.67 | 0.32 | <.001 | 1.37 | 1.41 |
|
|
Telemedicine intensity×time period | N/Aa | N/A | N/A | 2.48 | 0.64 | <.001 |
aN/A: not applicable.
Work outside work (WOW) per appointment by telemedicine intensity and time period.
Our study found that telemedicine was less efficient than in-person–based care and increased physicians’ WOW burden. The overall EHR-based WOW declined for physicians in the context of the COVID-19 pandemic and the rapid transition to telemedicine; however, when controlling for changes in patient volume and clinical hours of care, physicians who devoted a higher proportion of their clinical time to telemedicine had higher levels of EHR-based WOW than those who used telemedicine less intensively. This relationship was present during all phases of the study (before the pandemic, during acute pandemic, and after acute pandemic) and was amplified over time, including in the after acute pandemic phase. These findings suggest that the observed decrease in the average WOW during the pandemic was the result of the overall decrease in clinical load for physicians rather than any benefits or efficiencies of telemedicine itself. Further, the amplification of the relationship between WOW per appointment and telemedicine intensity in the time period beyond the acute pandemic suggests that the WOW increasing effect of telemedicine was exacerbated over time, and therefore, the unique circumstances of the early COVID-19 pandemic alone are insufficient to explain the behavior patterns of physicians.
There are several limitations to this study that future research could address. First, limitations in our Epic-based data set preclude the ability to review and analyze physician EHR activity with sufficient granularity beyond certain time periods; for example, time periods more specific than a calendar month or physician activity log data at smaller than 15-minute increments. Specifically, Epic does not count WOW in its time outside of scheduled hours if that work occurs within the 30 minutes before or after patient scheduled hours (a “shoulder period”), which our analysis is unable to reliably differentiate as WOW time and therefore excludes, resulting in a systematic underestimation of the true WOW. Moreover, because shoulder time is added for each clinical day regardless of length, this underestimation bias is greater for physicians who spread their patient time over more scheduled days relative to those who see the same number of patients on fewer days [
To our knowledge, this is the first study to systematically evaluate the impact of the transition to telemedicine during the COVID-19 pandemic on physicians’ after-hours workload and one of a few studies that used EHR-based data to objectively evaluate after-hours work burden [
A number of factors may be responsible for our findings that telemedicine increased the after-hours work burden of physicians. First, it is possible that organizational and technological inefficiencies in the early design, deployment, and scaling of telemedicine may have resulted in increased after-hours EHR work burden for physicians using telemedicine more intensively. These include early and ongoing technological issues relating to the computer hardware, software functionality and integrations, and user experience of the “virtual health” platform deployed by our system. These issues have been highlighted elsewhere in EHR and digital health technology implementation research, particularly regarding usability and user experience barriers [
The second factor that might have impacted our findings is that it is likely that significant disruptions to the work norms of clinical practices during the pandemic affected after-hours work patterns. In clinics, individual- and practice-level adjustments to the demands of care provision during the pandemic likely resulted in a number of unique work structures and arrangements that could have likely affected physicians’ work schedules, including time spent doing after-hours work. In particular, the shift to a telemedicine-based platform—particularly one with limited multiparty functionality—may have inhibited effective team-based care between physicians and clinical support staff (eg, medical assistants) and shifted both clinical and administrative tasks that had prior been completed by other staff members onto physicians. This “doctor does it all” phenomenon has been recently described as an unintended effect of the rapid transition to telemedicine during the pandemic [
Overall, our results suggest that telemedicine is not panacea for the work challenges facing clinicians. In fact, our evidence during the acute pandemic and after the acute pandemic suggests that rather than reducing administrative burden, telemedicine intensity may increase it, shifting the work temporally and spatially to after-hours work and home. This suggests that a more thorough understanding of the implications of telemedicine in clinical practice is necessary prior to its indiscriminate expansion to ensure policies and practices that increase efficiency and work-life quality and counteract inefficiencies, waste, and work-related stress and burnout are implemented. Given the limited data available on the impact of telemedicine on important aspects of physicians’ experience of work, it may be instructive to look to fields outside of medicine, where the study of “telework” (defined as a work arrangement that allows employees to perform work at approved alternative or remote worksites) [
In this study, we evaluated the impact of the transition to telemedicine during the COVID-19 pandemic on physicians’ EHR-based after-hours workload; we found that when controlling for the clinical load of patient visits, physicians who devoted a higher proportion of their clinical time to telemedicine engaged in higher levels of EHR-based after-hours work compared to those who used telemedicine less intensively; this relationship persisted and was amplified over time, even after the acute pandemic period. This suggests that telemedicine, as currently delivered, may be less efficient than in-person–based care and may contribute to after-hours work burden of physicians. Further study is needed on the detailed impacts of telemedicine on physician work practices, particularly in contexts beyond the COVID-19 pandemic and relating to administrative burden, after-hours clinical responsibilities (particularly the EHR-related in-basket and patient portal messaging responsibilities), and experience of work. Learning from other industries where telework is more established can help identify areas of need and opportunity in future telemedicine care delivery.
electronic health record
New York University Langone Health
pajama time
work outside work
We would like to acknowledge the work of our colleagues in the Department of Population Health. KL, ON, DM, and BW’s research is supported by the National Science Foundation (grants 1928614 and 2129076). The content of this work is solely the responsibility of the authors and does not necessarily represent the official views of the supporting agencies.
The data underlying this article will be shared upon reasonable request to the corresponding author.
KL, DM, ON, and BW significantly contributed to the conception and design of the study. EI and SM acquired the data. KL, BW, EI, and SM analyzed the data. KL, BW, ON, and DM drafted the initial manuscript. All authors were involved in data interpretation and manuscript revision and approved the final version submitted for publication.
None declared.