We performed a multicenter observational study to validate a computerized method of calculating CTPA yield. The study was conducted at two tertiary care hospitals, the North Shore University Hospital and the Long Island Jewish Medical Center in New York, in April and November, 2016, respectively. The two hospitals are supported by the Sunrise Clinical Manager electronic health record (EHR), a subsidiary of Allscripts Healthcare Solutions (Chicago, Illinois, United States). This study was approved by the Northwell Health’s Institutional Review Board.

The EHR databases at the two institutions are the replicated copies of the Sunrise Clinical Manager application. The database is replicated near real time with a <2-hour latency. Of note, this process is monitored by dedicated database administrators and analytics support team members to ensure fidelity. The databases were queried for CTPA orders completed in the ED over a 1-month period for each hospital. Patients’ visits were extracted from the EHR if they had a “completed” CTPA order during their ED course. However, patients with “cancelled” or “discontinued” CTPA orders were not included. Furthermore, patients with CTPAs ordered on the same day as CT angiography of the abdomen and pelvis were excluded, as these were under the protocol to rule out aortic dissection and not PE (Figure 1).

CTPA orders from the ED were then linked to inpatient visits. PE diagnosis was measured on the basis of an inpatient discharge diagnosis of the International Classification of Diseases, Clinical Modification codes, versions 9 and 10 (ICD-9-CM and ICD-10-CM), provided by the Centers for Medicare and Medicaid Services and the National Center for Health Statistics. We included both primary and secondary diagnoses in the analysis. Furthermore, the full range of PE diagnosis codes was used: 415.0, 415.11, 415.12, 415.13, and 415.19 for ICD-9-CM; and I26.0, I26.01, I26.02, I26.09, I26.9, I26.90, I26.92, and I26.99 for ICD-10-CM.

The CTPA yield was calculated as the number of ED CTPA orders linked to an inpatient discharge diagnosis of PE divided by the total number of CTPAs completed in the ED that month. This calculated yield was then validated by performing a manual chart review. In the manual chart review, the free-text radiology read of each completed ED CTPA order was reviewed to classify the CTPA as positive or negative for PE. In addition, both the ED provider note and inpatient discharge note were reviewed to ensure that the CTPA was done to evaluate for PE and the diagnosis was not revised during the inpatient visit.

The computerized calculated yield for each month and institution was compared with the yield generated from manual chart reviews by 2 independent reviewers. The reviewers were trained internal medicine physicians with experience in reading radiology reports. Of note, the reviewers were blinded to each other but not to the computerized results and had full access to discharge documentation and the entire medical chart.

We used McNemar’s test to determine whether the CTPA yields were different between the computerized calculated yield and the manual chart reviews. The kappa coefficient and the corresponding 95% CI were calculated to measure the agreement between the computerized calculated yield and the manual chart reviews.

To the best of our knowledge, this is the first study to propose and validate a simple, standardized method of electronically calculating the CTPA yield. This method has wide applicability to address increasing concerns about both overtesting and overdiagnosis of PE. The increase in the incidence of PE accompanying the increased use of CTPA [15] has been associated with a decrease in the PE case mortality [16,17]. Physicians are testing more for PE and seem to be finding and treating clinically insignificant PEs. The ability of health care systems to computerize the monitoring of the CTPA yield allows them to address overtesting and overdiagnosis using systems interventions.

In addition, clinical decision support tools, built to estimate the pretest probability of PE and discourage the CTPA use in low-risk patients, have been shown to improve the CTPA yield. These tools reduce testing by 25%, without any missed PEs [9,18,19]. However, these studies are limited by the time required for manual chart reviews. Studies of interventions designed to reduce unnecessary CTPA use decrease exposure to both contrast and radiation and avoid costly “incidentalomas.” Furthermore, incidental findings requiring clinical or radiological follow-up were found in 24% of patients without PE [16].

In this study, we describe a validated method to measure the CTPA yield that allows the data collection process to be computerized and does not require artificial intelligence. We utilized both ICD-9 and ICD-10 codes to fully encompass PE coding at the time of discharge. This method can be applied to allow for the comparison of the CTPA yield in different health care systems and different types of acute care facilities. Improved data collection will allow for more targeted interventions, with an ultimate goal of increased CTPA yields and decreased CTPA utilization.

The one discordant scan in our study points to a limitation that will likely become more relevant in future studies. One positive CTPA was missed by our computerized method because the patient was discharged directly from the ED, meaning there was no linkable inpatient visit or potential discharge ICD code. With a push toward cost-conscious care and away from inpatient medicine, there will likely be more patients with acute PE diagnosed in the ED who are treated as outpatients. While the safety of this practice was unclear and controversial just a few years ago [20,21], it has recently become more common with the increased use of direct-acting oral anticoagulants [22,23] and safety research in the field [24,25]. This is particularly true in large health care systems with tertiary EDs that can safely assess patients’ risk with bedside echocardiography and lower-extremity ultrasound [26]. Future studies will link CTPA scans to both inpatient and ED visits to improve the accuracy.

In addition, future studies may specify the type of PE and consider the discovery of a subsegmental PE as a negative study. This was not addressed in this study as currently, these are not treated differently and ICD codes do not distinguish these types of PEs. Notably, although this was not observed in our study, this method will likely count studies conducted in patients with chronic PE as positive. Finally, this study was conducted at two hospitals under one health care system, Northwell Health. Hence, future directions include studying this method at other institutions to ensure its accuracy.

This is the first successful validation study of a simple computerized method for calculating the CTPA yield in the ED. This method for data extraction allows for an accurate and efficient determination of the CTPA yield and represents a significant improvement from the manual chart review. With this ability, health care systems can monitor the appropriate use of CTPA and the effect of interventions to reduce overuse and decrease preventable iatrogenic harm.