In our study, we initially established a baseline model for classifying publications using traditional machine learning and word embedding techniques to demonstrate the effectiveness of employing a transformer-based model in identifying publications based on nested designs. To create the baseline model, we employed FastText (Facebook AI Research) to generate word embeddings, followed by a logistic regression model. Logistic regression has been widely recognized in the literature as an effective classifier for text data due to its simplicity, interpretability, and robust performance across various domains [23]. To enhance the model’s capability to process biomedical text data, we leveraged pretrained FastText embeddings specifically trained on PubMed and MIMIC-III data, known as BioWordVec [24], to extract meaningful features from titles and abstracts of publications. Subsequently, we used these extracted embeddings to train a logistic regression model for the classification of publications.

There are various metrics available to evaluate the performance of a classifier. While the area under the curve is commonly used in binary classifiers, accuracy can be helpful for balanced evaluation datasets. However, in our case, where the model serves as a multiclass classifier with imbalanced data, the F₁-score emerges as the most reliable metric [25]. Our primary objective was to fine-tune an existing pretrained language model to maximize the F₁-score on a validation dataset.

The F₁-score is the harmonic mean of precision and recall. Precision is the ratio of correctly predicted positive observations to the total predicted positives. It measures the accuracy of the positive predictions made by the model. High precision ensures that when our model predicts an article belongs to a nested design group, it is highly likely to be accurate. This is crucial in applications where precision contributes to the trustworthiness of the classification outcomes, such as systemically classifying research publications. Recall, which is the same as sensitivity, is vital when the cost of false negatives is high. A high recall ensures that our model effectively captures a comprehensive set of articles within each predefined group.

The F₁-score is particularly useful in situations where there is an imbalance between the classes or when there is an equal importance placed on precision and recall. It is a metric that balances the trade-off between precision and recall, providing a single value that reflects the overall performance of the model.

Our dataset is imbalanced with fewer examples in the IRGT and SWGRT classes, and accuracy alone can be misleading. Precision, recall, and F1-score provide a more nuanced view of a model’s effectiveness, especially in identifying the strengths and weaknesses associated with false positives and false negatives.

Specificity serves as a crucial metric for assessing the number of false positives within a model. It measures the ability of the model to correctly identify negative instances, thus providing insight into the model’s performance in avoiding false-positive predictions. Although it was not initially used as an evaluation metric during our experimental phase and model development, it was calculated to provide a comprehensive assessment of performance, particularly in gauging the rate of false positives.

The weighted average considers the class imbalance in the dataset by considering the contribution of each class proportional to the number of instances in that class. In other words, classes with more instances have a greater impact on the overall metric than classes with fewer instances.

Where

mi is the metric value (eg, precision, recall, F1-score) for class, ni is the number of instances in class i, C is the number of classes, and N is the total number of instances in the dataset

We chose BioMedBERT as the initial pretrained model due to its superior performance compared to other existing models on our gold standard data, publications published prior to 2021 that were identified as GRT, IRGT, and SWGRT papers using search queries. BioMedBERT is a pretrained language model developed by Microsoft Research for biomedical text processing using abstracts from PubMed and full-text articles from PubMedCentral [22]. It is a specialized variant of the BERT architecture [16], designed to capture domain-specific nuances in biomedical literature. The architecture of BioMedBERT enables it to learn contextualized representations of words and phrases bidirectionally and understand the contextual relationships within biomedical texts. The BERT model used in our study was downloaded from Hugging Face’s Transformers library, configured as a classifier. In this framework, the tokenizer automatically manages special tokens such as [CLS] and [SEP], ensuring proper preprocessing for input sequences.

For the classification task, the [CLS] token's embedding from the final layer of BERT serves as a representation of the input sequence's contextual information. A multilayer perceptron is applied to this embedding to perform the classification. This multilayer perceptron consists of fully connected layers that map the [CLS] token's representation to the output space, followed by the softmax activation for probability distribution over classes.

The pretrained BioMedBERT uses a bidirectional transformer architecture with several layers of self-attention mechanisms. The model embeddings, including word embeddings and positional embeddings, contribute to encoding patterns in biomedical language.

During the pretraining phase, BioMedBERT was initialized with weights obtained from training on a domain-specific corpus: 14 million abstracts, 3.2 billion words, and 21 gigabytes [22]. This large-scale training ensures that the model captures a wide range of biomedical concepts, terminology, and contextual relationships. BERT-BASE [16] with 12 transformer layers and 100 million parameters was used to pretrain BioMedBERT.

A pooling layer was introduced atop the transformer’s final layer, known as the embedding layer. This embedding layer underwent pooling to derive a fixed-size representation of the entire input sequence. To capture nonlinearity and intricate patterns within the data, a feedforward layer was incorporated. This layer is linked to the output layer, responsible for computing logits.

For this study, we fine-tuned BioMedBERT to adapt the model to the specific nuances of our dataset related to publications from clinical trials that use nested designs. By leveraging the contextual information encoded in BioMedBERT, we aimed to enhance the accuracy and efficiency of our machine learning model in identifying and distinguishing various types of clinical trial publications. In particular, our goal was for BioMedBERT to serve as a multiclass classifier that can categorize biomedical publications into four distinct categories: GRT, IRGT, SWGRTs, and the broader category of publications based on studies that used other designs, which we refer to as negatives (Multimedia Appendix 1).

The National Institute of Health Office of Disease Prevention provided a labeled dataset consisting of publications from PubMed, published prior to 2021, with each publication categorized into one of the three classes: GRT, IRGT, and SWGRT. We selected nonclinical trial publications from a list of 120 journals that published most of the nested clinical trial publications; we will refer to those publications as negative publications. The original dataset consisted of 891 GRT publications, 59 IRGT publications, 109 SWGRT publications, and 996 negative publications. The first version of the fine-tuned language model underwent training on titles and abstracts from this dataset. The best-performing model was subsequently employed to classify unlabeled publications published in 2021. After thorough verification of predictions by domain experts, a complementary set of 299 GRT, 40 IRGT, 65 SWGRT, and 1200 negative examples from 2021 was added to the original training, validation, and test sets.

The new dataset served as the foundation for tuning the model hyperparameters to predict labels for publications published in 2022. With the improved training data, the model demonstrated higher accuracy and F1-score, enabling precise classification of publications from the subsequent year, 2022. The same strategy as above was employed to add 2022 verified data to our training dataset, which resulted in the addition of 461 GRT, 195 IRGT, and 93 SWGRT, and 539 negative publications to the training, validation, and test datasets to prepare the classifier to predict labels for publications published in 2023. The Results section describes the final performance of the model trained and evaluated on this dataset.

To process the textual data, we utilized the Hugging Face Trainer API in conjunction with the BioMedBERT tokenizer. The BioMedBERT tokenizer is trained on a corpus of biomedical text to tokenize and segment text into subwords using the WordPiece algorithm, just like the original BERT tokenizer. Therefore, the tokenizer is tailored to handle biomedical terminology and language patterns [22]. For instance, when tokenizing the title “Comparison of different intervention methods to reduce the incidence of venous thromboembolism: study protocol for a cluster-randomized, crossover trial” from a 2023 publication, the tokens are segmented as follows: [“comparison,” “of,” “different,” “intervention,” “methods,” “to,” “reduce,” “the,” “incidence,” “of,’ “venous,” “thromboembolism,” “:,” “study,” “protocol,” “for,” “a,” “cluster,” “-,” “randomized,” “,”, “crossover,” “trial,” “.”]. The tokenization process was carried out separately for the titles and abstracts of the publications. In order to maintain computational efficiency and manage memory constraints, we imposed a length limit on the tokenized text. Titles were truncated to a maximum length of 30 tokens, which was the maximum title length in the data, while abstracts were truncated to 256 tokens from the start of the text, ensuring that majority of abstracts in our dataset fit within this allocation. Shorter sequences were padded with zeros. Based on the length distribution of titles and abstracts, we adjusted the allocation to align with actual usage patterns, ensuring that all instances remain within the supported range while optimizing model efficiency.

An imbalanced dataset refers to any dataset where there is an unequal distribution among classes, with one or more classes having significantly fewer instances than others. When trained on imbalanced datasets, models may exhibit a bias towards predicting the majority class more frequently, resulting in poor generalization for the minority class. Therefore, selecting the appropriate evaluation metric becomes crucial in such scenarios. A model might achieve high accuracy by predominantly predicting the majority class while displaying poor performance on the minority class. Various strategies exist to address class imbalance, including selecting appropriate performance metrics, such as precision-recall or the F₁-score, to accurately reflect model performance. Techniques like undersampling the majority class, oversampling the minority class, employing synthetic data generation methods like SMOTE (synthetic minority over-sampling technique), leveraging algorithms designed to handle class imbalance robustly, and incorporating cost-sensitive learning by assigning costs to the loss function are among the approaches commonly employed [26-28]. Given the observed class imbalance within the dataset, we implemented a customized loss function to mitigate the impact of this imbalance during the training phase. To achieve this, inverse class weights were calculated based on the number of examples in each class within the training dataset. These weights were then utilized in the weighted categorical cross-entropy loss function, assigning varying levels of importance to each class during model training.

The weighted categorical cross-entropy loss function is defined as follows:

The inverse class frequency method is used to assign higher weights to classes with fewer examples. The rationale is to give higher weights to classes that are under-represented, making the model more sensitive to minority classes and potentially improving performance on imbalanced datasets.

To evaluate the model’s performance, we split our dataset into three subsets: a training set, a validation set, and a test set. The test set comprised 20% of the total dataset and was created by stratified random sampling from the labeled data. To assess the generalizability of our model to unseen data and to ensure a robust and unbiased evaluation of our model, a stratified k-fold cross-validation technique was used by splitting the remaining 80% of the dataset into subsets to train and validate the model iteratively, k times. In this technique, the dataset is divided into k folds while maintaining the same class distribution in each fold as the original dataset. We chose k=5 for a 5-fold cross-validation. Each fold maintained the proportion of class labels similar to that in the overall dataset. This helped prevent the model from being biased toward the majority class [29].

We conducted a series of experiments to determine the optimal hyperparameters for our model. These included exploring various values for learning rates, weight decay, batch sizes, and the number of training epochs [22]. Our goal was to identify the combination of hyperparameters that produced the best model performance with the highest F1-score. The initial hyperparameters were selected based on the pretrained BioMedBERT model, and they were changed during the training process iteratively. The final model hyperparameters include a batch size of 32 and a learning rate of 0.00003098.

A regularization term is added to the loss function during training to penalize large weights and prevent overfitting [29]. The modified loss function with weight decay is calculated as follows:

Where original loss is the loss without regularization, λ is the weight decay hyperparameter, and Σi‖θi‖2 represents the sum of squared weights across all layers of the model.

The value of the weight decay hyperparameter is a crucial aspect of training. It determines the strength of the regularization effect. Too small a value may not prevent overfitting, while too large a value may penalize weights too much and stop the learning process. Moreover, early stopping was applied to the classifier head to prevent overfitting and improve efficiency with a patience of 5 epochs.

By following these steps, we developed a fine-tuned BioMedBERT model capable of classifying publications into the specified categories. This model was rigorously trained, validated, and optimized to maximize the classification accuracy and F₁-score.

After generating features using a FastText model and training the logistic regression model, we evaluated its performance across all classes. While the model demonstrated good performance on the majority class, indicating its efficacy in capturing prevalent patterns within the dataset, its performance on the minority class, IRGT, and SWGRT is suboptimal. Table 1 presents the baseline model’s performance across all classes, while Table 2 depicts the confusion matrix of the baseline model. The weighted average for precision, recall or sensitivity, specificity, and F₁-score is 0.85.

The low performance of the pretrained BioMedBERT classifier on all classes justifies the fine-tuning of the transformer-based model. Following the fine-tuning process using a training set that was curated by domain experts iteratively, the model’s performance exhibited noticeable enhancement in all performance metrics. The inclusion of domain expertise in the data curation process contributed to refining the model’s understanding and, consequently, improving its predictive capabilities.

Table 3 shows the performance of the latest version of the fine-tuned model on our test set, which was not seen by the model during training and validation. The confusion matrix, depicted in Table 4, provides a breakdown of the model’s predictions against the actual values. The weighted average for precision, recall or sensitivity, specificity, and F₁-score are 0.94.

The model developed in this research used data provided by the Office of Disease Prevention at the National Institute of Health and leveraged a transformer-based pretrained language model, BioMedBERT, to identify publications of clinical trials that used one of three nested designs. The model outperformed the baseline model developed using features generated by BioWordVec and logistic regression. Compared to our baseline model, the recall and specificity for each class demonstrated improvements as follows: 2 points for recall and 3 points for specificity in non-randomized class; 13 points for recall and 0 points for specificity in GRT; 36 points for recall and 1 point for specificity in IRGT; 26 points for recall and 1 point for specificity in SWGRT. The IRGT class exhibited the lowest sensitivity compared to other classes, due to fewer examples in the training data and the inherent difficulty in identifying such publications by only processing titles and abstracts. Even for human curators, labeling these publications is challenging, often necessitating meticulous examination of the whole paper’s content. Since the fine-tuned model was trained solely on titles and abstracts, it had limited information available for predicting IRGT publications.

While machine learning and natural language processing techniques have been utilized to identify RCTs in the medical literature [15], there have been fewer models specifically designed to identify special categories of group randomized clinical trials. Existing models for biomedical document classification predominantly rely on static embedding techniques, such as Word2Vec or FastText, although recent approaches have increasingly adopted nonstatic embedding methods, particularly transformer-based models like BERT and BioBERT, for more dynamic and context-aware text representations [30-32]. Our model leverages attention mechanisms and dynamic embedding techniques to capture the varying importance of words within the context of each document. By dynamically adjusting the embedding representations based on the context of the input sequence, our model can better capture the nuances and semantic relationships within the text, leading to improved classification performance.

Moving forward, future research endeavors may focus on refining the model to distinguish between subcategories within each main category, such as the method, protocol, and results [33]. Tailoring the model to address these distinctions could further enhance its utility in biomedical text classification tasks, facilitating more precise and comprehensive literature analysis.

Our study presents a robust framework leveraging transformer-based language models to effectively identify distinct categories of clinical trial publications within the biomedical literature. Through fine-tuning the pretrained BioMedBERT model, we achieved high accuracy and F₁-score metrics across three categories: GRTs, IRGTs, and SWGRTs. The developed framework outperforms conventional search queries, providing advanced language understanding capabilities for discerning a broader spectrum of publications.

Our findings underscore the significance of transformer-based models in biomedical text classification, offering improved performance compared to traditional machine learning approaches and static embedding techniques. By continually updating and refining our model with new training data, we anticipate ongoing improvements in performance and adaptability over time. This iterative approach ensures the model remains up to date on the latest developments in the biomedical field, contributing to more efficient literature exploration, information retrieval, and knowledge discovery.