This study was conducted at Asan Medical Center (AMC), a tertiary hospital in Seoul, South Korea. AMC is the largest medical center in Korea, with approximately 2700 inpatient beds and 10,000 outpatient visits per day. AMC has been fully accredited by the Association for the Accreditation of Human Research Protection Program since 2013; the IRB at AMC has received accreditation from the Forum for Ethical Review Committees in the Asian and Western Pacific Region since 2006.

In the process of clinical trials, AMC follows the US Food and Drug Administration regulations, as well as the Korean Good Clinical Practice and International Conference on Harmonization guidelines for AE reporting, which stipulate that all SAEs must be immediately reported to the sponsor except for those that the protocol or Investigator’s Brochure identifies as not requiring immediate reporting [12,13]. Sponsors are also required to report fatal and life-threatening suspected unexpected serious reactions (SUSARs) to regulatory agencies within 7 days after the sponsor’s initial receipt of the information, while other SUSARs may be reported within 15 days.

To manage clinical trials, we implemented a site-specific CTMS in December 2014 [10]. A CTMS is a system for managing clinical trials and research data. The requirements of various organizations related to clinical trials were developed for 14 months and implemented for 12 months based on the Java-based Spring Framework 4.0 (Pivotal Software). The CTMS was developed as an all-in-one system that links hospital information systems, the electronic IRB, enterprise resource planning, and biomaterial management systems in the respective academic medical center. The AEAS was implemented in July 2015 as a submodule of the CTMS interfaced with the EMR to facilitate early awareness of AE occurrences by investigators and study coordinators. The detailed operation process is as follows: if a patient makes an unscheduled visit to the hospital, the EMR confirms that the patient is in a clinical trial; if the patient participates in a specific clinical trial, the patient’s information is sent to the CTMS through the application programming interface; the CTMS then sends a text notification to the principal investigator or clinical research coordinator regarding the patient’s emergency room visit or hospitalization.

From December 2014 to June 2016, 196 clinical trials were managed through the AMC CTMS. Among these trials, 93 had at least one AE report, with a total of 305 initial, 403 follow-up, and 179 finalized AE reports. In this study, we included 305 initial AE reports from 74 clinical trials to measure the improvement in AE awareness by the implementation of the AEAS. The follow-up or finalized reports were excluded from this study because they could not serve to verify the effectiveness of the AEAS in matters such as early recognition of AEs.

A total of 117 initial AEs were processed in the CTMS prior to the implementation of the AEAS—December 2014 to June 2015—and 188 initial AEs were processed postimplementation—July 2015 to June 2016 (see Figure 1). Overall, patient AE monitoring processes were the same in the pre- and postinitiative periods, but the AE awareness method was different: people-dependent versus system-supported. Of the 74 clinical trials that were evaluated, 41 were carried out during the preinitiative period and 53 during the postinitiative period, with 20 trials overlapping the two periods.

The following data were collected: the dates of initial AE onset when investigators became aware of a given AE and when the AE report was submitted. The intervals between each step were calculated, with AE onset defined as the time when patients made an unscheduled visit to the emergency room or clinics.

General characteristics were compared using chi-square tests. The numbers of days between AE reporting phases (ie, onset, awareness, and reporting) were presented as medians with IQRs and mean ranks with SDs among the periods before and after AEAS implementation. The Shapiro-Wilk normality test showed the time intervals between AE onset, awareness, and reporting to have nonnormal distributions. We, therefore, used the Wilcoxon rank-sum test for comparing the pre- and postinitiative periods. We also presented the effect size of a pre- and postcomparison with a Cohen d test. According to US Food and Drug Administration guidelines, AEs are reported as being on business days [14-16]; likewise, we set the intervals using business days. We also performed subgroup analyses according to risk level and study phase. The risk level of each clinical trial was determined based on the classification by the ADApted MONitoring (ADAMON) project [17]. A higher risk level value means a subject has a higher than lower risk level. P values lower than .05 were deemed statistically significant. All analyses were performed in R, version 3.5.3 (The R Foundation).

This study was approved by the IRB of AMC (IRB No. 2015-1368). The need for informed consent was waived by the ethics committee on the basis that this study utilized routinely collected medical data that were anonymously managed at all stages, including data cleaning and statistical analyses.

A total of 305 initial AE reports from 74 clinical trials were included in this study (see Table 1). Most initial AE reports subjected to analysis occurred in sponsor-initiated trials (286/305, 93.8%) and multisite trials (273/305, 89.5%). Initial AE reports were more common in phase 3 trials, followed by phase 1 and 2 trials; approximately 70% of initial AE reports were from risk-level 3 trials (210/305, 68.9%). Most of the basic characteristics of the clinical trials were not significantly different between the pre- and postinitiative periods. The frequency of global trials was higher in the postinitiative period.

All three AE awareness performance indicators were significantly lower in the postinitiative phase (see Table 2 and Figure 2). However, due to the skewed distribution (see Figure 2), median values were 0 days in both the pre- and postinitiative phases. Statistical testing found the distributions to be significantly lower, and examination of the mean rank demonstrated that reporting times were lower and more consistent overall in the postinitiative phase. In particular, the onset to reporting showed a statistically significant difference (preinitiative: median 1 day, mean rank 173.82, vs postinitiative: median 1 day, mean rank 140.04, P<.001). The onset-to-awareness and awareness-to-reporting time also showed significant reductions between the pre- and postinitiative periods (preinitiative: median 0 days, mean rank 164.15, vs postinitiative: median 0 days, mean rank 146.06, P=.04; and preinitiative: median 1 day, mean rank 173.82, vs postinitiative: median 1 day, mean rank 140.04, P=.02, respectively).

Figure 2 shows the date difference between pre- and postinitiatives by three AE awareness indicators. For all three AE awareness indicators, the phenomenon of AE records taking more than 25 business days preinitiative disappeared postinitiative.

The phase subgroup analysis showed that all AE awareness indicators were reduced from the preinitiative period (see Table 3). The awareness-to-reporting and onset-to-reporting indicators in phase 1 studies showed statistically significant differences in median values and mean rank pre- and postinitiative (preinitiative: median 1 day, mean rank 47.94, vs postinitiative: median 0 days, mean rank 35.75, P=.01; and preinitiative: median 1 day, mean rank 47.4, vs postinitiative: median 1 day, mean rank 35.99, P=.03, respectively). In phase 4, there was no statistically significant difference in the onset-to-reporting timeline, but the SD of postinitiative ranks was significantly smaller than that of preinitiative ranks (preinitiative: rank SD 16.41 vs postinitiative: rank SD 11.89). Phase 3 and 4 studies showed greater differences than did phase 1 and 2 studies in the pre- and postinitiative SD values of all AE awareness performance indicators.

In the risk-level subgroup analysis, decreases of all AE report timelines were observed between pre- and postinitiative periods (see Table 4). Onset to awareness and onset to reporting of low-risk trials (level 1) showed significant reductions in the median and mean rank (preinitiative: median 4 days, mean rank 18.91, vs postinitiative: median 0 days, mean rank 11.65, P=.02; and preinitiative: median 4 days, mean rank 18.73, vs postinitiative: median 1 day, mean rank 11.76, P=.02, respectively). In high-risk trials (level 3), the distributions of awareness to reporting and onset to reporting were significantly different pre- and postinitiative (preinitiative: median 0 days, mean rank 115.94, vs postinitiative: median 0 days, mean rank 98.26, P=.02; and preinitiative: median 1 day, mean rank 117.36, vs postinitiative: median 1 day, mean rank 97.27, P=.01, respectively).

The most notable finding of this study is that the timeline from patients’ unscheduled visits (ie, onset), due to AE, to safety reporting was significantly reduced after the implementation of the AEAS. Also, the variability in the amount of time between patient visits and investigator awareness was lower after the implementation. This suggests that the AEAS notifications are effective for improving the speed with which investigators or clinical research coordinators are informed, thereby allowing them to take prompt action against AE occurrences.

There have been several approaches to enhance the efficiency, completeness, and consistency of safety reporting through the use of Web-based electronic safety reporting modules [18,19]. However, such approaches do not detect patients’ unscheduled visits at the site level. To our knowledge, this is the first study to evaluate the effectiveness of the implementation of an IT-driven AE awareness system for clinical trials at the site level. Early safety signal awareness with the alert system may contribute to better patient protection after the occurrence of AEs. A recent study found that many SAE results registered in ClinicalTrials.gov were yet to be published or omitted from publications [20]. Due to the imbalance of information on AE reporting, there is a high demand for more comprehensive approaches to ensure the safety of clinical trial participants [20-22]. There are only a few reports that have evaluated the timeline between AE onset and the initial reporting of the AE at the site level. One study reported that the mean duration from onset to reporting was 25 days for nonserious AEs and 11 days for SAEs [18]. However, to the best of our knowledge, there are no reports on other systems designed to improve the detection of AEs during clinical trials. Caution is necessary when comparing our findings with those of other studies because of the differences in protocols, safety reporting systems, and sites. However, our improved safety reporting metrics—from unscheduled visit to reporting—which were less than 2 days in the postinitiative period, deserve highlighting.

Another interesting finding is that the improvement in turnaround times after AEAS implementation was more prominent in phase 1 and in low- and high-risk clinical trials. The primary goal of a phase 1 study is to identify safety profiles and determine the dose-limiting toxicities of a new drug. This might be due to the increased attention being paid to participants in phase 1; therefore, the improvement in the reporting timeline was not as noticeable in trials of other phases as it was in phase 1 trials. We note that the safety reporting timeline is shorter postinitiative than preinitiative in all phases of clinical research.

The main limitation of our study is that our study results are based on a retrospective analysis of clinical trials carried out in a single academic medical center. Also, this system could not detect patient visits to other hospitals, which would require patient self-reporting; the rate of patient self-reporting may be improved with education, telephone monitoring, and questioning during trial visits. Another limitation is that we only analyzed the timeline for reporting AEs. Nevertheless, detailed analyses of AEs in clinical trials were not accessible at this point, as such analysis is only possible when the data are reported to the regulatory agency and published. Due to the study’s retrospective nature, there is a trend of imbalances among several characteristics, such as phase of trial or number of sites.

In this study, we demonstrated that the AEAS, a CTMS-driven real-time automatic alert system, can effectively improve safety reporting timelines. The AEAS resulted in overall reductions in AE awareness timelines in all clinical trials, especially in phase 1 trials and low- and high-risk studies. These findings suggest that IT-driven automatic alert systems are effective in improving safety reporting timelines, which may ultimately enhance patient safety.