A Computerized Method for Measuring Computed Tomography Pulmonary Angiography Yield in the Emergency Department: Validation Study

Introduction

The ability of computed tomography (CT) to diagnose pulmonary embolism (PE) was demonstrated in 1980 [1]. The introduction of multidetector row CT pulmonary angiography (CTPA) revolutionized the diagnostic approach to PE in 1998 [2]. The availability and use of this new technology rapidly increased in the following years, and by 2001, CT overtook the ventilation/perfusion lung (V/Q) scan as the most common method for diagnosing PE [3].

In 2006, results from the landmark prospective investigation of pulmonary embolism diagnosis (PIOPED) II trial established CTPA as the first-choice diagnostic imaging modality, with a sensitivity of >90% for patients with high clinical suspicion of PE and a specificity of 96% [4,5]. Over the next 5 years, there was a 4-fold increase in CTPA use and a 33% decrease in V/Q scanning [6]. However, CTPA is associated with a nearly 7-fold higher radiation burden than V/Q scanning [6], with attributable lifetime malignancy risk of up to 2.76% in young female patients [7]. Moreover, in a recent prospective study, it was found that up to 14% of patients who underwent CTPA developed contrast-induced nephropathy [8].

Increased rates of CTPA use and improved understanding of the associated adverse effects have prompted researchers to measure the CTPA yield [9-11]. The CTPA yield is a measure of the appropriateness of use, defined as the percentage of tests completed to evaluate for PE that are positive for PE. The majority of these studies have used manual chart abstraction to calculate the CTPA yield [9-11]. Furthermore, a form of artificial intelligence, natural language processing, has been shown to reliably calculate the CTPA yield in a few recent studies [12-14].

These methods have demonstrated reliability but are time consuming or require technology not available at most health care institutions. To date, a simple, standardized method of electronically calculating the CTPA yield has not been described. The objective of this study is to propose and validate a computerized method for calculating the CTPA yield in the emergency department (ED).

Methods

Procedure

We performed a multicenter observational study to validate a computerized method of calculating CTPA yield. The study was conducted at two tertiary care hospitals, the North Shore University Hospital and the Long Island Jewish Medical Center in New York, in April and November, 2016, respectively. The two hospitals are supported by the Sunrise Clinical Manager electronic health record (EHR), a subsidiary of Allscripts Healthcare Solutions (Chicago, Illinois, United States). This study was approved by the Northwell Health’s Institutional Review Board.

The EHR databases at the two institutions are the replicated copies of the Sunrise Clinical Manager application. The database is replicated near real time with a <2-hour latency. Of note, this process is monitored by dedicated database administrators and analytics support team members to ensure fidelity. The databases were queried for CTPA orders completed in the ED over a 1-month period for each hospital. Patients’ visits were extracted from the EHR if they had a “completed” CTPA order during their ED course. However, patients with “cancelled” or “discontinued” CTPA orders were not included. Furthermore, patients with CTPAs ordered on the same day as CT angiography of the abdomen and pelvis were excluded, as these were under the protocol to rule out aortic dissection and not PE (Figure 1).

CTPA orders from the ED were then linked to inpatient visits. PE diagnosis was measured on the basis of an inpatient discharge diagnosis of the International Classification of Diseases, Clinical Modification codes, versions 9 and 10 (ICD-9-CM and ICD-10-CM), provided by the Centers for Medicare and Medicaid Services and the National Center for Health Statistics. We included both primary and secondary diagnoses in the analysis. Furthermore, the full range of PE diagnosis codes was used: 415.0, 415.11, 415.12, 415.13, and 415.19 for ICD-9-CM; and I26.0, I26.01, I26.02, I26.09, I26.9, I26.90, I26.92, and I26.99 for ICD-10-CM.

The CTPA yield was calculated as the number of ED CTPA orders linked to an inpatient discharge diagnosis of PE divided by the total number of CTPAs completed in the ED that month. This calculated yield was then validated by performing a manual chart review. In the manual chart review, the free-text radiology read of each completed ED CTPA order was reviewed to classify the CTPA as positive or negative for PE. In addition, both the ED provider note and inpatient discharge note were reviewed to ensure that the CTPA was done to evaluate for PE and the diagnosis was not revised during the inpatient visit.

The computerized calculated yield for each month and institution was compared with the yield generated from manual chart reviews by 2 independent reviewers. The reviewers were trained internal medicine physicians with experience in reading radiology reports. Of note, the reviewers were blinded to each other but not to the computerized results and had full access to discharge documentation and the entire medical chart.

Data Analysis

We used McNemar’s test to determine whether the CTPA yields were different between the computerized calculated yield and the manual chart reviews. The kappa coefficient and the corresponding 95% CI were calculated to measure the agreement between the computerized calculated yield and the manual chart reviews.

‎View this figure

Results

In total, 375 CTPAs were completed during the allotted period for review. Of them, 6.9% (26/375) orders were completed on the same day as a CT angiography of the abdomen and pelvis and were, thus, excluded. Furthermore, manual chart reviews confirmed that each of these 26 omitted CTPA orders and none of the remaining 349 were ordered with the intention to rule out an aortic dissection. There were no cases where a chart review of the ED provider note and inpatient discharge note changed the interpretation of the CTPA results. In addition, the independent reviewers reported the same results, with no disagreement.

At the North Shore University Hospital site, of 203 CTPA orders completed for the evaluation of PE, 18 orders were found to have an associated inpatient discharge diagnosis of PE. The calculated yield was 8.9% (18/203). Manual chart reviews revealed 19 positive scans for a true CTPA yield of 9.4% (19/203). Notably, one discordant scan was found in a patient directly discharged from the ED, and as a result, the patient never received an inpatient discharge diagnosis.

At the Long Island Jewish Medical Center site, 146 CTPA orders were completed and 9 were found to have an associated inpatient discharge diagnosis of PE. The calculated CTPA yield was 6.2% (9/146). The manual chart reviews produced identical results, confirming 9 positive CTPA scans.

Overall, the computerized method captured 27 of 28 scans positive for PE, with an accuracy of 96.4% (27/28; Figure 2). The overall CTPA yield for both institutions was 7.7% (27/349). In this study, the P=.32, indicating that the proportions were not significantly different between the two groups. Furthermore, the kappa coefficient was .98, with 95% CI (0.94-1.00) also indicating an agreement between the two groups.

‎View this figure

Discussion

Principal Findings

To the best of our knowledge, this is the first study to propose and validate a simple, standardized method of electronically calculating the CTPA yield. This method has wide applicability to address increasing concerns about both overtesting and overdiagnosis of PE. The increase in the incidence of PE accompanying the increased use of CTPA [15] has been associated with a decrease in the PE case mortality [16,17]. Physicians are testing more for PE and seem to be finding and treating clinically insignificant PEs. The ability of health care systems to computerize the monitoring of the CTPA yield allows them to address overtesting and overdiagnosis using systems interventions.

In addition, clinical decision support tools, built to estimate the pretest probability of PE and discourage the CTPA use in low-risk patients, have been shown to improve the CTPA yield. These tools reduce testing by 25%, without any missed PEs [9,18,19]. However, these studies are limited by the time required for manual chart reviews. Studies of interventions designed to reduce unnecessary CTPA use decrease exposure to both contrast and radiation and avoid costly “incidentalomas.” Furthermore, incidental findings requiring clinical or radiological follow-up were found in 24% of patients without PE [16].

In this study, we describe a validated method to measure the CTPA yield that allows the data collection process to be computerized and does not require artificial intelligence. We utilized both ICD-9 and ICD-10 codes to fully encompass PE coding at the time of discharge. This method can be applied to allow for the comparison of the CTPA yield in different health care systems and different types of acute care facilities. Improved data collection will allow for more targeted interventions, with an ultimate goal of increased CTPA yields and decreased CTPA utilization.

Limitations

The one discordant scan in our study points to a limitation that will likely become more relevant in future studies. One positive CTPA was missed by our computerized method because the patient was discharged directly from the ED, meaning there was no linkable inpatient visit or potential discharge ICD code. With a push toward cost-conscious care and away from inpatient medicine, there will likely be more patients with acute PE diagnosed in the ED who are treated as outpatients. While the safety of this practice was unclear and controversial just a few years ago [20,21], it has recently become more common with the increased use of direct-acting oral anticoagulants [22,23] and safety research in the field [24,25]. This is particularly true in large health care systems with tertiary EDs that can safely assess patients’ risk with bedside echocardiography and lower-extremity ultrasound [26]. Future studies will link CTPA scans to both inpatient and ED visits to improve the accuracy.

In addition, future studies may specify the type of PE and consider the discovery of a subsegmental PE as a negative study. This was not addressed in this study as currently, these are not treated differently and ICD codes do not distinguish these types of PEs. Notably, although this was not observed in our study, this method will likely count studies conducted in patients with chronic PE as positive. Finally, this study was conducted at two hospitals under one health care system, Northwell Health. Hence, future directions include studying this method at other institutions to ensure its accuracy.

Conclusions

This is the first successful validation study of a simple computerized method for calculating the CTPA yield in the ED. This method for data extraction allows for an accurate and efficient determination of the CTPA yield and represents a significant improvement from the manual chart review. With this ability, health care systems can monitor the appropriate use of CTPA and the effect of interventions to reduce overuse and decrease preventable iatrogenic harm.