Chronic obstructive pulmonary disease (COPD) is a leading cause of mortality in the United States, impacting an estimated 24 million people nationwide [1]. COPD exacerbation–related hospitalizations also incur substantial health care costs, ranging from US $7000 to US $39,200 per hospitalization [2]. Overall, COPD-attributable care amounted to US $49 billion in 2020, with hospitalization comprising approximately half of this total cost [3].

Artificial intelligence (AI) is transforming health care by enhancing the analysis, interpretation, and prediction of medical data. This technological advancement supports clinical decision-making and enables more efficient allocation of limited resources for chronic disease management, particularly for high-risk patients [4,5]. While narrative clinical text is information rich, its unstructured nature poses challenges for direct use in AI-driven clinical decision support systems. To address this, information extraction and representation methods have emerged, leveraging large language models (LLMs) and deep learning techniques [6-10]. These approaches aim to transform raw clinical narratives into structured, machine-readable data. However, significant challenges persist when processing vast document collections. Information relevant to a specific task may be dispersed across only a small fraction of the total corpus, making it inefficient to apply computationally intense LLMs to the entire dataset [11]. Sophisticated preprocessing, filtering, and prioritization algorithms can identify and extract relevant documents, reducing the volume of text that needs to be processed at computationally intense stages of the pipeline [12].

Weak supervision is a machine learning (ML) approach that mitigates the challenges of creating large, annotated datasets for natural language processing (NLP). It involves using lower-accuracy or less detailed programmatically generated labels and limited labeled data in semisupervised learning [13-15]. This approach reduces the need for extensive manual labeling, speeding up the ML process and cutting costs. This method offers several advantages, including reduced annotation costs, improved scalability, and enhanced domain adaptability [13-15]. Weak supervision allows for the incorporation of domain expertise, supports iterative refinement, and enables model development in low-resource scenarios [13-15]. By minimizing manual labeling efforts, it facilitates the efficient creation of large-scale, domain-specific datasets, making it particularly valuable for developing robust NLP models across diverse fields [16-18].

Several recent studies have demonstrated the effectiveness of weak supervision in medical NLP tasks. Wang et al [19] combined this approach with pretrained word embeddings to create a rule-based NLP algorithm for the automatic generation of training labels. They evaluated the effectiveness of these auto-generated labels across 4 supervised learning architectures: support vector machine, random forest (RF), fully connected networks, and convolutional neural networks. Their comparative analysis demonstrated the versatility and efficacy of this auto-labeling technique as the generated labels performed successfully across diverse model frameworks [19]. Similarly, Cusick et al [18] applied weak supervision to address the challenge of exhaustive manual labeling in clinical contexts. Building on a previous study that used a rule-based NLP system to automatically label the clinical notes of 600 patients with potential suicidal ideation [18], they efficiently generated a sizable training dataset. This dataset was then used to train various statistical ML models and a convolutional neural network. Their research highlighted the potential of combining weak supervision and deep learning to enhance real-time clinical systems and facilitate research on suicidal ideation progression through automated analysis of clinical text [18]. In another study in this field, Fries et al [16] introduced Trove. This framework for weakly supervised medical entity classification leverages existing medical ontologies such as the Unified Medical Language System (UMLS) as a source of reusable, automated labeling heuristics. Trove’s key innovation is its use of a label model to learn the accuracies of individual ontologies and correct for label noise when combining multiple sources. The authors demonstrated a weakly supervised performance on named entity recognition tasks for chemicals, diseases, and drugs that achieved results within 1.3 to 4.9 F₁-score points of fully supervised models.

These studies underscore the growing importance of weak supervision in overcoming the limitations of traditional manual annotation methods in clinical NLP tasks. By enabling the efficient creation of large-scale, domain-specific datasets, weak supervision is proving to be a valuable approach for developing robust NLP models across diverse clinical applications. This method offers a simplified way to auto-generate training labels, significantly reducing the need for manual annotation [16-19].

Our proposed method uniquely integrates high-accuracy expert labels with weak supervision, offering a more efficient and accurate clinical document classification solution that overcomes traditional techniques’ limitations. In the following section, we will describe the rationale, significance, and unique contributions of our proposed approach, demonstrating how it advances clinical NLP methods and addresses critical needs in the field.

NLP models designed for categorizing clinical documents can support various downstream AI or ML tasks in health care, such as risk prediction, clinical summarization, and probabilistic phenotyping [20-22]. The recent introduction of LLMs has created tremendous potential for health care to leverage these powerful tools. However, the bottleneck lies in the vast number of documents that these computationally intensive models need to process, which can reduce their efficiency and effectiveness. Lightweight NLP models solve this challenge by efficiently filtering documents and providing appropriate data inputs for more sophisticated models such as LLMs [10,23-25]. These lightweight models could potentially identify and select relevant documents, which may enhance the accuracy and efficiency of the overall AI or ML pipeline. However, the effectiveness of these models depends on input data quality. International Classification of Diseases (ICD) codes, traditionally used to identify relevant documents, have proven ineffective [26,27]. This observation underscores the limitations of relying exclusively on ICD codes for document selection in this context. The findings suggest the necessity for more refined and thorough identification methods to ensure the accurate capture of relevant COPD-related documents.

The potential effectiveness of these lightweight NLP models could stem from their ability to accurately identify and filter out irrelevant documents while potentially maintaining efficiency in computing power, memory use, and processing time. This efficiency is especially advantageous for resource-constrained health care systems, where simpler ML approaches that require significantly less computational power are often preferred over advanced NLP techniques [28] such as LLMs.

By leveraging heuristic training data and a small expert-annotated dataset, these models can be optimized with minimal labeled data, significantly reducing the manual annotation burden. This approach addresses the persistent challenge of limited annotated data in clinical settings, potentially accelerating the development and deployment of AI-driven health care solutions [19,29].

Our study makes 3 primary contributions to the field of clinical document classification. First, we introduce a novel framework that uniquely integrates high-accuracy expert labels with weak supervision for hyperparameter tuning, thus designing lightweight ML classifiers. This approach offers a more efficient and accurate solution for categorizing COPD-related documents as either “relevant” or “nonrelevant,” overcoming the limitations of traditional techniques. Second, we leveraged heuristic training data formulated by domain-expert clinicians to construct a training dataset without the need for extensive manual annotation. By using these expert hypotheses to generate “silver-standard” labels [30,31], we substantially mitigated the challenges commonly associated with manual annotation. Third, we conducted extensive experiments with diverse text representation techniques, rigorously evaluating the relative performance of different pipelines for identifying relevant documents while minimizing the inclusion of irrelevant ones.

Although our study focused on patients with COPD, the techniques we developed are designed to be adaptable across various clinical domains. Our approach of combining a small number of high-accuracy expert labels for hyperparameter tuning with a larger set of lower-accuracy labels for model training allows for potential improvements in model performance that may not be achievable using only lower-accuracy labels.

We constructed a retrospective observational cohort of patients with COPD attending Vanderbilt University Medical Center (VUMC) between January 1, 2012, and December 31, 2020. A series of NLP pipelines of all clinical narrative notes from this cohort were used to perform data evaluation. The objective of this study was to create an AI or ML pipeline using weak supervision with a small set of annotated data that could effectively determine the relevance of the notes for future AI or ML tasks for a clinical condition of interest (COPD). Our cohort inclusion criteria were patients with a diagnosis of COPD at any point during the study period and who were aged 40 years or older at the time of their diagnosis. COPD was defined based on administrative codes available from the electronic health record (EHR) for the ICD, Ninth Revision, Clinical Modification (491, 491.1, 491.2, 491.21, 491.22, 491.8, 491.9, 492, 492.8, 493.92, and 496) or ICD, Tenth Revision, Clinical Modification (J41.0, J41.1, J41.8, J42, J43.1, J43.2, J43.8, J43.9, J44.0, J44.1, and J44.9).

Figure 1 illustrates our NLP framework for classifying clinical notes related to COPD, including heuristic data creation, text representation, model training, optimization, and evaluation. Our heuristic training data, or “silver standard,” consisted of 20,000 clinical notes. We created this dataset using expert-defined rules: notes from encounters with COPD diagnosis codes were labeled as informative (positive), whereas those from encounters without COPD diagnosis codes or from 24 months or more prior to initial COPD diagnosis were labeled as noninformative (negative). This heuristic approach enabled the creation of a large training dataset without manual annotation, leveraging domain expertise to generate plausible labels. We used various text representation techniques to transform the free text into numerical features, including bag of words (BoW), term frequency–inverse document frequency (TF-IDF), lightweight document embedding, compression-based embeddings, and UMLS concept extraction. These representations fed into 3 ML classifiers: RF [32], k-nearest neighbor (KNN) [33], and extreme gradient boosting (XGBoost) [34]. For model optimization and evaluation, we used a separate gold-standard dataset manually annotated by clinical experts. We used a subset of these data for hyperparameter optimization, allowing us to fine-tune our models based on high-quality, expert-labeled data, and a held-out test set for final model assessment. By leveraging a large heuristically labeled dataset for training and then optimizing and testing with a carefully curated gold standard, this approach integrates heuristic-based training data creation, diverse text representation techniques, and a 2-stage evaluation process using expert-annotated data. The following subsections will explain each block in more detail.

This study was approved by the VUMC Institutional Review Board (IRB #210139) and received a waiver of informed consent as retrospective secondary use of routinely collected data. Data were stored and analyzed on secure, encrypted on-premises servers with restricted access. All study team members with access to data were trained in human participants research ethics. No participant compensation was provided. The manuscript and supplementary materials present summary information only and contain no identifiable patient information.

The defined ICD-based heuristics were evaluated against manually annotated gold-standard test data. Table 1 shows the baseline model’s performance in classifying COPD-related documents. Figure 2 shows the baseline confusion matrix on the test data. The heuristics correctly identified 50 irrelevant and 57 relevant documents while misclassifying 24 irrelevant and 92 relevant documents.

The results validate the effectiveness of our framework, which used various text representation techniques and ML models to eliminate irrelevant documents and prioritize information in relevant ones. We found that combining both negative cases (non-COPD coded and pre-COPD temporal) consistently produced the best model across all representation techniques. As shown in Tables S1 to S5 in Multimedia Appendix 1, the mixed-negative configuration yielded the highest F₁-score across most text representations, supporting this choice as the primary training strategy. Tables S1 to S5 in Multimedia Appendix 1 provide comprehensive results for all combinations, but here, we focus on the results obtained by training the model on the combined dataset using both negative heuristics.

Table 2 presents a comparative analysis of various ML models (RF, XGBoost, and KNN) across different document representations. The RF model consistently demonstrated superior performance with balanced precision, recall, and F₁-score for the relevant class when using BoW representation. In the TF-IDF representation, KNN showed high precision but lower recall, whereas RF provided a more balanced performance. With lightweight document embeddings, RF achieved high recall but lower precision for the irrelevant class, indicating a trade-off. For compression-based representations, RF and XGBoost performed comparably, with KNN achieving a higher F₁-score. Finally, in the UMLS representation, XGBoost marginally outperformed RF in precision and F₁-score, whereas KNN showed high precision but lower recall.

Lightweight document embedding combined with the RF model was the most effective method for extracting COPD-relevant documents. This approach achieved an F₁-score of 0.80 (95% CI 0.76-0.87) for the relevant class, outperforming other representation techniques examined in this study.

The baseline model confusion matrix (Figure 2) correctly identified 50 irrelevant and 57 relevant documents, with 24 irrelevant and 92 relevant documents misclassified. The best-performing NLP pipeline confusion matrix (Figure 3) correctly identified 27 irrelevant and 128 relevant documents, with 47 irrelevant and 21 relevant documents misclassified. The best-performing model significantly improved relevant document identification (128 vs 57) and reduced relevant document misclassification (21 vs 92).

Although the best-performing NLP pipeline had a higher irrelevant document misclassification rate (as highlighted by low specificity), this trade-off was acceptable given the optimization for sensitivity, which was the primary goal. The detailed results for the best-performing classifier at various cutoff points are shown in Figure S1 in Multimedia Appendix 1. The figure illustrates the relationship among precision, sensitivity, and specificity across different thresholds for the classification model.

Our study demonstrated the effectiveness of lightweight ML models, particularly an RF classifier with lightweight document embeddings, in extracting COPD-relevant documents from EHRs. This approach significantly outperformed baseline heuristic methods, achieving high recall (0.86) and F₁-score (0.80, 95% CI 0.76‐0.87) for relevant documents. These findings represent a significant advancement in efficient EHR processing, offering a scalable solution for filtering large volumes of clinical documents in resource-constrained environments. By improving the identification of relevant information, this approach has the potential to enhance the performance of clinical decision support systems and optimize resource use in health care applications.

Traditional heuristic approaches, such as relying on ICD codes for document filtering, have proven inadequate in accurately capturing relevant clinical information. Our findings confirm that a substantial proportion of documents with COPD ICD codes lack significant relevant information (as illustrated in Figure 2), highlighting the limitations of such methods. In contrast, our ML framework effectively distinguished relevant from irrelevant documents (as illustrated in Figure 3), thereby improving the sensitivity and identification of relevant documents. ML models outperformed baseline approaches by optimizing small annotations, learning deeper patterns, adapting flexibly, generalizing better from imperfect examples, handling noisy data robustly, and using advanced feature extraction techniques [16,47].

A primary contribution of our work is demonstrating how lightweight models such as an RF classifier paired with lightweight document embeddings can achieve high performance with minimal annotation data. By using weak supervision, we generated “silver-standard” labels from domain-expert heuristics, significantly reducing the need for extensive manual annotation. This method not only streamlined the training process but also enhanced the model’s scalability across different clinical domains.

Our methodology uniquely integrated a large corpus of weakly supervised silver-standard labels for initial training with a small set of high-fidelity gold-standard labels for hyperparameter optimization. This approach not only streamlined the training process but also enhanced model performance and generalizability across diverse clinical domains.

As shown in Table 2, among the evaluated text representation techniques, lightweight document embedding coupled with the RF classifier exhibited superior performance in classifying relevant COPD documents, achieving a recall of 0.86 and an F₁-score of 0.80 (95% CI 0.76-0.87) for the relevant class. This performance significantly surpassed the baseline model’s F₁-score of 0.50 (95% CI 0.43-0.56), underscoring the efficacy of our approach compared to conventional heuristic methods.

The incorporation of a limited set of high-fidelity labels for hyperparameter tuning resulted in a substantial improvement in model performance. The model-agnostic nature of this approach rendered it applicable to any classifier requiring hyperparameter optimization.

Additionally, the results underscore the critical role of text representation in model performance. While both RF and XGBoost proved to be robust classifiers, their effectiveness was notably influenced by the choice of text representation technique. For instance, RF’s F₁-score for the relevant class ranged from 0.57 (95% CI 0.52-0.60) with compression-based representation to 0.80 (95% CI 0.76-0.87) with lightweight document embedding.

While our study demonstrated the effectiveness of lightweight document embedding and ML models in extracting relevant COPD documents, several limitations should be considered. First, the dataset used in this study was specific to COPD, and the generalizability of our findings to other clinical conditions remains to be evaluated. Future research should examine the applicability of this framework across a broader range of diseases and clinical document types. Second, the data were derived from a single institution (VUMC), which may limit external validity due to variations in clinical documentation practices across health care systems. Multi-institutional validation using datasets from different health systems would help assess the robustness and portability of the proposed framework. Finally, while this study focused on binary classification of documents as relevant or irrelevant, future work could explore more granular categorization of clinical documents through multi-class classification (eg, exacerbation-related notes, diagnostic documentation, and treatment updates). Such extensions could further enhance the utility of this framework for downstream clinical decision support and LLM-based information extraction pipelines.

This study presents a novel framework for extracting COPD-relevant clinical documents using lightweight document embedding and ML models. Our approach effectively identifies relevant documents while minimizing the identification of irrelevant ones, enhancing the quality of information for clinical decision support systems and improving patient outcomes. Future research should explore the generalizability of our findings to other chronic conditions and integrate this model into predictive analytics for greater efficiency and effectiveness. This framework can be adopted across diverse health care domains, contributing to improved patient care and clinical outcomes.