This study was based on the processing and analysis of quality control (QC) documents from a large number of clinical trials, which underwent their own ethics reviews. Further, the text of the QC findings did not contain personal information of trial participants. As such, this study did not require ethics board review.

A total of 1591 QC reports were generated in our institution from January 2019 to September 2023, involving 993 clinical trials. In accordance with relevant quality management standard operating procedures, a specific QC group was responsible for conducting at least 3 rounds of routine QC for each trial, including but not limited to (1) an early-stage QC round at the time point of achieving 5 enrollments or at 3 months after trial initiation, (2) an interim-stage QC round once every 6 months or when the trial achieved 20 enrollments, and (3) a conclusion-stage QC round at the time of site closeout. A detailed QC report was generated for each QC round, recording all quality issues. To build a digital monitoring platform, 1380 reports, which were generated during 2019 to 2022, served as the foundation for automated quantitative processing and the identification of warning thresholds, while the remaining 211 reports, which were generated in the year 2023, were used for testing the effectiveness of those thresholds in identifying high-risk trials.

Findings from the routine QC reports were rated by using 3 severity grades (minor, major, or critical) and classified into 5 categories (“Data,” “Quality,” “Safety and Ethics,” “Equipment and Facility,” or “Finance”), with 4 taxonomy levels under each category. The severity ratings and Level-1 to Level-3 taxonomies would allow for an overall review and trend analysis, while the Level-4 taxonomies were designed to trace back precisely to the corresponding clinical trial procedure and personnel.

To enable the automated classification of QC findings, we developed a natural language processing (NLP) algorithm based on the Local Interpretable Model-Agnostic Explanations framework, which was implemented in Python. This algorithm was designed to perform multidimensional classifications of events in QC report texts (Figure 1).

The dataset used to develop the NLP algorithm comprised 1380 QC reports that were generated during 2019 to 2022; an average of 345 reports were generated per year. Among QC findings from these reports, 55.48% were categorized as minor findings, 42.23% were categorized as major findings, and 2.29% were categorized as critical findings. The word count of the reports ranged from 1619 to 17,367 words, with a median word count of 5938 (IQR 4453-7422) words per report. Data were stratified and split into training, validation, and testing sets (70%, 15%, and 15%, respectively) to ensure balanced class representation. Annotation rules covered 4 entity dimensions and 54 issue categories, which were derived from industry standards and refined through discussions with an internal expert team.

In the first phase, the dataset was annotated by using multidimensional information labeling rules, generating high-quality training data. In the second phase, natural text from QC reports was subjected to preliminary automated classification via a Bayesian network, which determined the Level-1 to Level-3 classifications for each text segment based on joint probability. Additionally, semantic role labeling—a critical step in the classification NLP tool—was used to identify the predicate-argument structure of each sentence [8]. After training with the dataset, an explainable directed acyclic graph was constructed [9]. In the directed acyclic graph, graph neural networks were used to propagate information across the graph, forming an algorithmic model capable of capturing events. The model achieved precision (P) and recall (R) scores >0.95. In the third phase, by using the trained model, events were extracted from the QC report text and subjected to Bayesian inference. If an inference could be drawn from the graph, automatic classification was completed; if no inference could be made, a new branch was created, and the entire graph model was updated and iterated. The final Level-4 classification results were represented as the terminal leaf nodes in the graph.

After all findings from QC reports were classified by using our automated classification tool, a dataset was created based on the incidence rates of different categories of QC findings from each QC report. Using SPSS for Windows 20.0 (IBM Corp), hierarchical clustering analysis was applied to this dataset to ensure that reports with similar patterns of QC findings were grouped into the same cluster: each report was initially treated as a separate cluster; then, the algorithm identified 2 clusters that were closest to each other (ie, clusters with maximum similarity) and merged these 2 clusters into 1 new cluster. These steps were repeated until all QC reports were grouped into 2 clusters (Cluster I and Cluster II). A dendrogram was generated, showing the levels of similarity and dissimilarity between QC reports. Per manual interpretation by a group of experienced QC analysts, Cluster II was identified as high risk (ie, reports with more findings or a higher proportion of major and critical findings), and Cluster I was considered relatively “normal” (reports with less findings or a higher proportion of minor findings).

Warning thresholds were established based on the high-risk (Cluster II) QC reports from 2019 to 2022; the lowest number of findings was set as the threshold value for a specific study stage, Level-3 taxonomy, and severity grade combination. For example, a threshold value of X was set for the following combination: early-stage QC round, Level-3 taxonomy “Biological Sample Management,” and major findings; this made sure that all high-risk QC reports from early-stage QC rounds had at least X major findings under the Level-3 taxonomy “Biological Sample Management.” The warning thresholds were then applied to QC reports from 2023; these warning thresholds would help with determining the risk points (ie, the most commonly reported findings) and investigating the root causes of findings.

From 2019 to 2022, the proportion of major or critical findings generally decreased each year (major findings showed a 1.27-fold decrease, and critical findings showed a 3-fold decrease), even during the COVID-19 pandemic, while the proportion of minor findings showed an approximately 1.24-fold increase over the 4 years (Figure 2).

It was consistently observed during each year that the most commonly reported Level-3 taxonomies (the proportions of findings for these taxonomies were >5%), in decreasing order of frequency, were (1) “Standard Procedure and Process,” (2) “Safety Reporting,” (3) “Source Data Collection and/or Recording,” (4) “Investigational Product,” (5) “Personnel Qualification and Training,” and (6) “Biological Sample Management” (Table 1). In 2020, under the influence of the COVID-19 pandemic, increases were observed in the average numbers of findings per report that fell under these Level-3 taxonomies, but 2 years later, the average numbers of these findings dropped back to their 2019 levels (Figure 3).

Similar trends were also observed when QC findings were summarized under Level-4 taxonomies. A 1.41-fold increase in the average number of findings under the Level-4 taxonomy “Adverse Event Assessment and Documentation” was observed in 2020 when compared to 2019. Further, while the average number of findings under the Level-4 taxonomy “Biological Sample Shipment and Recording” was ≤0.2 per report across each year, a 1.43-fold increase was also noted in 2020 when compared to the previous year. These trends were likely attributable to the outbreak of the COVID-19 pandemic. With the adaptive actions taken in response to COVID-19 (eg, implementation of digital monitoring), the numbers of QC findings under these two Level-4 taxonomies eventually dropped back to their 2019 levels (Figure 4).

A total of 189 warning threshold values were established to cover all 3 severity grades, 21 Level-3 taxonomies, and 3 rounds of QC (Multimedia Appendix 1). These thresholds were applied to a total of 211 QC reports generated in 2023. Among those reports, 42 triggered warnings, resulting in a trigger rate of 19.9%. Approximately half (n=23, 55%) of the 42 warning-triggering, high-risk reports were from interim-stage QC rounds, while 8 and 11 were from early-stage and conclusion-stage QC rounds, respectively.

Figure 5 presents data from 3 randomly selected reports from early-stage QC rounds as examples to show the number of major QC findings that exceeded or did not exceed the corresponding threshold value for each of the 21 Level-3 taxonomies.

Among the 42 high-risk reports in 2023, after the application of warning threshold values, the most frequently noted Level-3 taxonomies with alarm-triggering QC findings were generally consistent with those observed in QC reports from 2019 to 2022 (Tables 1 and 2, Figure 3). The most common alarm-triggering Level-3 taxonomy was “Standard Procedure and Process,” which was reported in 36% (15/42) of high-risk reports, and its most relevant process (ie, “risk point,” as defined by Level-4 taxonomies) was “Protocol Deviation – Deviation from Study Design,” which was reported in 33% (5/15) of the reports that triggered an alarm for QC findings that fell under the Level-3 “Standard Procedure and Process” taxonomy. In response to Level-4 taxonomies that were more likely to cause alarm-triggering QC issues (Table 2), intervention strategies to be incorporated into routine clinical practice were proposed.