Three Obstetrics-Gynecology (Ob-Gyn) scenarios were created by a clinical subject matter expert (Ob-Gyn attending physician) to represent standard ambulatory encounters. We created a replica of each clinical case in our simulation instance of EpicCare (Epic Systems) using techniques we have described in previous publications [4,18]. Briefly, the EHR instance utilized for simulation activities is created from a “clone” of the clinical system, maintaining all user customizations, shortcuts, and macros. The instance contains only patient charts representing the simulation; it does not contain protected health information of real patients in our health care system. Given the need for any simulation-based training exercise to be both scalable and accessible from a variety of clinical environments, we decided to use a virtual video-based simulation. For each scenario, we video-recorded a standardized patient-provider encounter, with medical personnel serving in roles of both physician and patient. Once recorded, each video was cropped and edited to ensure adequate audio and video quality. On the basis of the script of each scenario, a “gold-standard” note was created for each case to allow for assessment of accuracy of content of individual scribe notes.

A list of all medical scribes was collected from the Scribe Program Supervisor of the OHSU medical scribing program. Medical scribes working at the OHSU Center for Women’s Health (CWH) were selected because they represented the largest proportion of all medical scribes working at OHSU. They were approached via email, phone texts, and phone calls to arrange simulation participation times. All scribes had a minimum of 1 year of scribe experience and minimum 6 months of experience scribing for CWH before study participation.

In order for the simulations to accurately replicate scribes’ work environment in real-world settings, the activity was conducted at the CWH, OHSU. For each simulated case, subjects were instructed to (1) familiarize themselves with each simulated patient chart before beginning the simulated physician-patient video, and (2) perform scribe activities in simulation just as they would during a real physician-patient interaction.. Simulations were performed in patient exam rooms at the CWH, OHSU that replicated real-world conditions accurately. Videos were displayed from a laptop computer on the exam table. Scribes used dedicated exam-room computers. The standardized narrative was read aloud to each scribe. Each simulation lasted between 6 and 18 min and scribes performed all three cases, in the same order.

Scribe- and physician-created notes were transferred from the Epic simulation environment into Pages (Apple Inc). Screenshots were taken of the Encounter, Labs, and Imaging tabs of Chart Review to determine whether the orders were pended. The gold-standard note was transferred from the Epic simulation environment into Pages in the same manner.

Scribe notes were evaluated for note length, word economy, data elements, copy and paste blocks, pended orders, and attestations. These structural elements were compared with each other to determine interscribe variability. Structural elements were also compared with our gold-standard note to determine accuracy and positive predictive value (PPV). PPV was defined as the ratio of scribe’s data elements also found in the gold-standard note to all those data elements included by the scribe. Data elements were defined as the individual positive and negative facts created by the scribe or gold standard from each of the patient-physician videos and provided resources. Data elements represented the interpretation of the scribe and the gold standard with respect to what was verbalized and performed during the encounter. Data elements were tabulated by note section, subjective, objective, or assessment and plan. The presence of copy and pasted blocks was determined using Plagiarism Checker X (Plagiarism Checker X, LLC), a plagiarism detection software package. Word economy was defined as the number of words required to create 1 data element or the number of words divided by data elements. Attestations were considered present if the medical-scribe included a statement at the end of their note signifying that they were a scribe working on behalf of the physician-provider.

In this study, we created a novel virtual simulation to specifically assess scribe use and function. The use of a standardized video encounter carries the distinct advantage of untethering the simulation from a traditional simulation center, thereby improving accessibility of the training activity to multiple clinical environments. This represents a more scalable alternative, given how scribes are already reported to work in a variety of clinical environments and are deeply embedded in community clinics, many of which may not have access to traditional simulation. In addition, the use of a standardized video ensures consistency of the delivery of content, allowing for direct comparison of work-product between scribes and across practices.

With the standardization of the delivery of content and inclusion of the EHR as an integral part of the simulation activity, we were able to allow direct interscribe comparisons between notes, which revealed significant variability in note structure and length. There is a lack of clarity with respect to the extent of experience medical scribes require to attain any particular level of competency. Despite the fact that all of the scribes had at least 1 year of experience both in the specialty and with the EHR, there was almost a 3-fold difference in note length. Even more interesting was the difference in actual “note” elements between scribes. This is consistent with findings from studies showing discrepancies between physicians in the content and quality of documentation in notes [19,20]. Thus, whereas this phenomenon is most likely not unique to scribes, it does imply that scribes may face the same issues that are found among other clinicians.

Although the simulation provides the basis to assess differences in note structure, we were also able to create a methodology to look at note content. We found evidence of errors of commission (incorrect data) and omission (missing data) by comparing the data elements found in notes written by scribes versus the notes written by an expert clinician. Notably, there was a paucity of overlap in content between the notes, with less than 40% of the documented plan items and diagnoses being common across the scribes. This is consistent with the observation that there is wide variability in the content of resident-physician-generated progress notes, where the primary author of the note (the resident) was also responsible for acquisition of the primary data and synthesizing that information into medical decision making [20]. This study suggests that similar issues may arise purely in the process of how our subjects communicate as members of an interprofessional team. However, this study does not delineate whether the differences observed are because of the individual scribe workflows, scribe deficits in medical knowledge, issues related to scribe training, or lacunae in scribe-physician communication. The use of a controlled simulated case may also explain the differences between our results and a recent study looking at actual scribe-generated notes in a practice setting [21]. In that study, scribe-generated notes for diabetes encounters, with medical assistants serving as scribes, created equally “readable” notes compared with physician-created notes. However, since each individual note corresponded to a unique patient encounter, there was no true “gold standard” for the information transmitted during that visit. This highlights the power of using simulation as an objective tool for determining competency, by controlling for the actual clinical content verbalized. Given the variability among scribe training and experience, their ability is likely also variable. Through the use of high-fidelity simulation exercises, one can standardize their training to ensure that all scribes reach objective benchmarks required for clinical practice.

It is important to note some important limitations to this study. Whereas this study focused on note creation, which is the primary role of the scribe, it did not address other scribe-specific activities such as data entry and data gathering [22,23]. Although we have previously demonstrated feasibility in integrating this into EHR-focused simulations, examining these other tasks will need to be the focus of future studies. Second, this study was a proof-of-concept study with a small number of scribes in a single specialty. Whereas the differences in note content and structure were noteworthy, a much larger cohort will be required to fully define the magnitude and scope of any potential safety issues in documentation and EHR usage. This is even more important, given the wide spectrum in baseline scribe training and prior experience in medical care before functioning as a scribe. Third, even though the simulations were designed to be easily deployed across multiple environments, additional studies will be required to determine the quantity and content of training required for novice educators (eg, providers) to access, deploy, and assess the work output from these activities, especially in community and rural settings. Finally, in real-world workflow, scribe notes must be attested and signed by an attending physician. Thus, it is unclear how much of the variance observed in the note structure would persist in actual clinical care after this final, attending physician–level vetting.

In conclusion, our study highlights the variability of scribe documentation and the need for a more standardized approach to training. This proof-of-concept study demonstrated a means of effectively evaluating scribe performance.