Negation and speculation detection in clinical text has progressed from rule-based algorithms [8-13] to modern deep learning–based approaches [1,3,4,14-21]. Early rule-based tools [8-13] used lexical patterns, with or without syntactic parsing, to define negation scope but struggled with text containing unusual grammar or requiring cross-sentence analysis. Machine learning methods [22-26] learned statistical cues from labeled corpora and outperformed rule-based models, yet remained constrained by handcrafted features and annotation demands.

With deep learning [14-17,19-21], convolutional neural networks learned salient local context, whereas recurrent neural network–based models (eg, long short-term memory and gated recurrent unit models) better captured sequential dependencies and broader semantic relationships. Recent studies [1,3,4,18] show that fine-tuned transformers achieve strong performance in negation detection; however, their performance may depend on large human-labeled datasets. For instance, our previous work [1] used 6000 annotated radiology reports, including 4500 training samples.

KD transfers the predictive behavior of a large “teacher” model to a smaller “student” model, enabling compact architectures to retain accuracy while reducing computational requirements [27,28]. Widely adopted in NLP, KD enables smaller transformers to approximate larger models by mimicking their output distributions or internal representations. For example, DistilBERT (Distilled version of Bidirectional Encoder Representations from Transformers) reduces BERT’s (Bidirectional Encoder Representations from Transformers) number of parameters by approximately 40% while retaining most of its linguistic capabilities [29].

The rise of LLMs extends KD into a new paradigm in which powerful models such as GPT-3 generate high-quality pseudo-labels without task-specific fine-tuning [7]. Transformers trained on these labels can approach human-labeled baselines while reducing annotation costs by up to 50%-96% [7], although LLM-derived labels remain less reliable than expert annotations [30].

Recent studies have introduced KD into clinical NLP, including the compression of clinical transformers [28] and the enhancement of information extraction from biomedical text [31,32] and public health communication [33]. However, none of these studies has systematically evaluated KD as a label-efficient supervision strategy for radiology report negation detection or directly compared few-shot LLMs with human-labeled transformers for this specific task.

We deidentified all included radiology reports by removing identifying information, including medical record numbers, application numbers, examination dates, ordering departments, and examination times.

We inherited the dataset from our previous study, in which the primary annotator (KHW) generated the original labels. For this study, we formalized the previous annotation guidelines without substantively changing the annotation criteria to enable independent annotation by a second annotator.

After finalizing the annotation guidelines, we randomly selected 1000 reports from our dataset to assess interrater agreement using Cohen κ.

Author BLH, a physician coauthor in postgraduate training, independently annotated the reports, while the primary annotator (KHW) reannotated the same cases. KHW and BLH calculated word-level agreement, resolved discrepancies by consensus, and revised the labels accordingly.

We refined the labels for negations and speculations unrelated to abnormal findings using our in-house software, which served as the gold standard for analysis. Annotators labeled spans throughout the entire report rather than only the predefined target keywords. We labeled each token as either category 0 (positive or speculative statements potentially related to abnormal findings) or category 1 (negative or speculative statements unrelated to abnormal findings), as in our previous study (Table 1) [1].

As in our previous work, we designed the annotation scheme to optimize downstream clinical workflows, including clinical alerts, keyword highlighting, and case retrieval, rather than general-purpose negation detection. Accordingly, we used a task-specific annotation scheme.

For our downstream task, explicit negations and speculations unrelated to abnormal findings both indicate the absence of disease presence or suspicion in radiology reports. We therefore annotated them as a single category to align with the downstream clinical objective.

For secondary analysis, we further divided category 1 tokens into 4 linguistic patterns but treated them as a single class during model evaluation. Following our previous work [1], we categorized negations as sentential, symbolic, or morphological. We identified speculation unrelated to abnormal findings in 2 situations: (1) when the radiologist explicitly denied a finding, for example, “No evidence of A; suggest exam B to rule out A if indicated,” or (2) when a diagnosis could not be evaluated or excluded because of study limitations, such as motion artifacts, technical issues, or suboptimal contrast.

We reshuffled the dataset and split it into training, validation, and test sets at a ratio of 7:1:2 (4200, 600, and 1200 reports, respectively). The validation set was used solely for early stopping during transformer fine-tuning.

We split the test dataset without enforcing patient-level separation, allowing follow-up studies from the same individuals to appear across subsets. This design reflects real-world practice, in which new data naturally include patients who undergo multiple examinations.

We also constructed an engineering dataset comprising approximately 80 training samples for iterative LLM prompt refinement.

Figure 2 summarizes the model development and KD framework.

We fine-tuned 3 transformer backbones across 28 training data sizes and 3 data-sampling seeds, using expert labels with random or stratified sampling or pseudo-labels generated by 3 LLMs through 10-shot prompting with random sampling. We also evaluated the LLMs through direct 0-, 5-, and 10-shot inference, using 3 example-selection seeds for the 5- and 10-shot settings. Figure 3 illustrates the keyword-level evaluation process for all models.

As shown in Figure 3, our experiment followed a structured 3-stage workflow to assess keyword-level filtering performance.

First, we removed negations and speculations unrelated to abnormal findings using both LLMs and transformers to process all test-set reports. Each model aimed to detect and remove tokens labeled as class 1 by experts.

Second, we performed text realignment. After filtering, we realigned each LLM output with the original report using the aforementioned 2-pass DP algorithm.

Third, we performed keyword matching and evaluation. Although annotators labeled spans throughout each report, we evaluated performance at the predefined keyword level because this evaluation reflects the intended clinical workflows, such as keyword highlighting and case retrieval. For each target high-priority keyword (bleeding, consolidation, dissection, extravasation, fracture, infarct, metastasis, pneumoperitoneum, pneumothorax, pulmonary TB, thrombosis, and tumor) [1], we determined, based on the prefiltered text, whether it occurred within a negated or speculative context unrelated to abnormal findings (y=1) and whether the model masked it (ŷ=1). We then computed keyword-level precision, recall, and F₁-scores and derived subtype-specific recall across 4 negation subtypes to assess model performance. All performance metrics in our study reflect keyword-level filtering performance for predefined clinical alert keywords, not general span-level negation detection, unless otherwise specified.

This study included 2 complementary analytical frameworks. Primary and secondary comparative analyses (comparison families A-G) used 10,000 paired bootstrap resamples of held-out test reports to estimate paired differences in F₁-scores and, for subtype-specific analyses, recall, along with 95% CIs and multiplicity-adjusted P values.

Descriptive performance and sensitivity analyses were conducted directly on the original test dataset without bootstrap resampling. These analyses included performance under maximal-data conditions, sensitivity analyses of patient overlap, and variability across data sampling seeds.

We performed all statistical analyses in Python 3.13.5 (Python Foundation). We set the significance level to α=.05 and used 2-sided tests unless stated otherwise.

Using a paired bootstrap framework, we compared the primary outcome (keyword-based F₁-score) and the secondary outcome (recall for each of the 4 negation subtypes).

We generated 10,000 bootstrap replicates by sampling with replacement from the 1200-report test dataset and computed group performance on the same set of replicates to ensure strict pairing across all comparison families.

For each contrast between groups A and B within a fixed training-size range, we defined the paired difference as Δr = metric_A,r − metric_B,r. The metrics for groups A and B (metric_A,r and metric_B,r, respectively) were averaged across all compatible seed-specific model configurations, including training data size, backbone architecture, label source, sampling strategy, and data seeds, as applicable.

The empirical distribution of Δr yielded the mean difference, the 95% percentile CI, and the 2-sided empirical P value: 2 × min(P[Δr≥0], P[Δr≤0]). We controlled for multiplicity using the Holm-Bonferroni procedure and rejected the null hypothesis (Δ=0) when the adjusted P value was <α. The 95% CIs were not adjusted for multiplicity.

In this study, we considered training sample sizes below 1000 to represent relatively small-scale transformer training [36]. We selected 840 as the cutoff, as it is close to 1000 and corresponds to 20% (840/4200) of the full training set, to distinguish between small- and large-sample conditions.

For analyses A, B, C, D, E, and G, primary inferences were made across the small-sample (n=42-840) and large-sample (n=1260-4200) ranges, followed by secondary exploratory analyses at each training size.

We conducted a post hoc descriptive analysis of residual limitations at the case level under best-case pseudo-label supervision. We selected the 10-shot LLM with the highest F₁-score as the teacher and, at the maximum training set size, identified the transformer architecture with the smallest F₁-score reduction under pseudo-label supervision relative to its human-supervised counterpart. Both models used the same backbone, 4200 training reports, and sampling strategy. We then manually reviewed test cases that the human-supervised transformer classified correctly but the pseudo-label–supervised transformer misclassified, along with the corresponding predictions from the LLM teacher.

We conducted a descriptive sensitivity analysis by excluding patients in the test dataset who also appeared in the training or validation sets and recalculating F₁-scores for models evaluated under maximal-data conditions. For each model configuration, F₁-scores were averaged across 3 data sampling seeds before and after exclusion, and the corresponding difference (ΔF₁-score) was reported. We did not perform bootstrap resampling or hypothesis testing for this analysis.

We conducted a descriptive sensitivity analysis of multiseed variability on the original test dataset without bootstrap resampling. We evaluated 5- and 10-shot LLM inference across 3 few-shot example selection seeds and all transformer configurations across 3 training-data sampling seeds used during fine-tuning. For each configuration, we calculated keyword-level F₁-scores for each seed and reported the mean and sample SD of the 3 seed-specific F₁-scores (Multimedia Appendix 2).

Table 5 summarizes 4 sensitivity analyses addressing assumed gains in annotation efficiency, patient overlap, model-specific prompt adaptation, and random seed selection. The following subsections report the corresponding quantitative results and methodological details.

Assuming no efficiency gain from labeling familiarity, GPT-5.3 Chat–assisted labeling (requiring 6-12 hours of prompt engineering) became less expensive than 126 hours of expert annotation at hourly wages above US $0.64-0.67.

We evaluated potential patient-level data leakage by reassessing the Table 4 models after removing 253 out of 1111 (22.77% in the test dataset) unique patients who overlapped with the training and validation datasets. ΔF₁-scores ranged from –0.002 to 0.006, with mixed directions, consistent with random variation and suggesting a negligible impact of patient overlap on performance evaluation.

The lightweight adaptation yielded only minor and inconsistent changes in performance (F₁-scores: Gemma 4, 0.878-0.858; Qwen3.6-Plus, 0.836-0.858). This analysis used only the engineering dataset (about 80 reports) under the zero-shot setting rather than 10-shot performance on the held-out test dataset. We developed and evaluated the adapted prompts using the same engineering dataset; therefore, the evaluation was not independent of prompt development.

We evaluated 1260 transformer seed-specific configurations, comprising 420 base configurations defined by training set size, backbone, label source, and sampling strategy, each evaluated using 3 training data sampling seeds.

Keyword-level F₁-scores showed low but nonnegligible variability (Multimedia Appendix 2). On the full test set, the median SD was 0.012 (IQR 0.005-0.022; range 0.000-0.077). Small-sample configurations (n=42-840; 300 base configurations) showed greater variability (median SD 0.015; IQR 0.008-0.025) than large-sample configurations (n=1260-4200; 120 base configurations; median SD 0.006; IQR 0.003-0.012).

Most base configurations showed limited seed-related variability: 295 of 420 (70.2%) base configurations had an SD ≤0.020, calculated from unrounded values. However, a minority, predominantly those with smaller training sizes, showed greater variability. The largest SD occurred for ALBERT trained on Qwen3.6-Plus–distilled labels with n=252 under random sampling (mean F₁-score 0.897; SD 0.077). Overall, transformer performance remained generally stable across seeds, although some configurations showed substantial seed-related variability, especially in the small-sample regime.

Across the 3 evaluated LLMs, observed SDs were numerically higher with 5-shot prompting (median 0.010; range 0.003-0.021) than with 10-shot prompting (median 0.007; range 0.006-0.008).

Across training sizes, human-labeled transformers outperformed those trained on LLM-distilled labels, with the largest gap at small sample sizes (n=42-840); this gap narrowed but persisted up to n=4200 (see Figure S1A in Multimedia Appendix 4). Post hoc analyses confirmed that all distilled students (GPT-5.3 Chat, Gemma 4, and Qwen3.6-Plus) underperformed their human-labeled counterparts (see Figure S1A, post hoc analyses, in Multimedia Appendix 4), indicating that the performance loss was independent of the teacher model. Overall, KD yielded lower performance than expert supervision, and larger datasets mitigated but did not eliminate the limitations of LLM-generated labels.

Transformers trained on distilled labels from GPT-5.3 Chat, Gemma 4, and Qwen3.6-Plus showed overlapping performance across training sizes (see Figure S1B in Multimedia Appendix 4), with per-N differences fluctuating around zero and no consistent teacher advantage (Multimedia Appendix 3). This pattern mirrored the primary comparative analysis, which found no significant differences among distilled transformers under either small-sample conditions (adjusted P>.99 for all 3 pairwise comparisons) or large-sample conditions (GPT-5.3 Chat vs Gemma 4, adjusted P=.88; GPT-5.3 Chat vs Qwen3.6-Plus, adjusted P=.47; Qwen3.6-Plus vs Gemma 4, adjusted P=.88). Although GPT-5.3 Chat performed best in the bootstrap evaluation (comparison family F), its distilled students did not outperform those trained on labels from other teachers, suggesting that interteacher differences were attenuated after distillation.

Stratified and random sampling produced overlapping performance curves across training sizes (see Figure S2C in Multimedia Appendix 4), with per-N differences centered around zero (Multimedia Appendix 3). Stratified sampling showed minor, inconsistent advantages at small training sizes that did not persist across data points. Overall, the sampling strategy had little effect on F₁-scores under expert supervision, whereas training data size appeared to play a more substantial role.

At small training sizes, student models achieved slightly lower F₁-scores than their teachers. As training size increased, the performance gap progressively narrowed, and student models achieved performance comparable to that of their teachers across all 3 LLMs (see Figure S2D in Multimedia Appendix 4 and its zoomed view). Overall, teacher-student differences remained modest and diminished further with larger datasets (Multimedia Appendix 3).

Across training sizes, human-labeled transformers showed a steady upward performance curve (see Figure S3 in Multimedia Appendix 4). At small sample sizes, they underperformed GPT-5.3 Chat but quickly surpassed Gemma 4 and Qwen3.6-Plus.

In large samples, they consistently outperformed Gemma 4 and Qwen3.6-Plus and converged toward GPT-5.3 Chat, with closely overlapping curves and small differences. These results agree with the primary comparative analysis, which showed larger performance advantages over Gemma 4 (adjusted P<.001) and Qwen3.6-Plus (adjusted P<.001) and a small but statistically significant advantage over GPT-5.3 Chat (adjusted P=.047) in the large-sample range.

As LLMs were evaluated under a fixed 10-shot configuration, their performance was independent of transformer training size (see Figure S3 in Multimedia Appendix 4) and therefore did not warrant per-N secondary analysis.

Sentential negation, the dominant pattern in the test set (n=982), showed almost identical recall curves under stratified and random sampling (see Figure S3 in Multimedia Appendix 4). Morphological negation was absent from the test dataset (n=0), precluding the calculation of recall. Symbolic or telegraphic negation (n=169) varied moderately, without consistent separation between sampling strategies. Speculation unrelated to abnormal findings (n=9) fluctuated substantially, and these small counts warrant caution in interpretation. Overall, stratified sampling did not show a statistically significant recall advantage for sentential negation (adjusted P>.99 under both small- and large-sample conditions) or symbolic or telegraphic negation (adjusted P=.16 under small-sample conditions and P>.99 under large-sample conditions). P values were not calculated for pseudo-speculation or morphological negation.

We grouped speculative statements unrelated to abnormal findings with explicit negations because both have the same operational consequence in our downstream applications: neither should trigger keyword highlighting or clinical alerts or be returned by a case-finding application designed to identify positive or potentially positive cases. Thus, we defined labels according to the downstream clinical objective rather than conventional linguistic taxonomy.

We do not propose this task-specific scheme as a universal standard; institutions may tailor annotation policies to local workflows. Transformer-based models can accommodate such policies through retraining on locally annotated data, whereas LLM-based approaches can use institution-specific prompts and few-shot examples without retraining. Therefore, our study emphasizes that annotation policies may be tailored to clinical objectives rather than that our scheme should be strictly followed.

Our keyword-based metrics may differ from those used for general-purpose negation detection. Although we trained the models on span-based annotations, we primarily evaluated whether they correctly filtered predefined clinical alert keywords. General-purpose evaluations at the character, token, word, or span level may instead penalize discrepancies in negation scope boundaries. Thus, our high keyword-level performance should not be extrapolated to tasks requiring exact scope agreement. For example, the negation spans “No US evidence of thrombosis at bilateral lower extremities” and “No US evidence of thrombosis” both correctly identify the clinically important term “thrombosis” as negated but may receive different token- or word-level scores depending on their agreement with the expert-annotated scope.

Even with an identical annotation scheme and evaluation framework, institution-specific differences, such as note types, authors, and linguistic characteristics, may cause domain shifts and degrade model performance [39]. We therefore expect cross-institutional variation in radiology reporting styles, formats, terminology, and radiologist-specific writing patterns to affect model performance.

Institutions may benefit from fine-tuning transformer-based negation detection models on locally annotated data to accommodate differences in annotation schemes and linguistic distributions simultaneously. This need also highlights the practical value of our KD framework, which can efficiently generate large-scale pseudo-labels for training institution-specific models or for subsequent expert refinement, provided that task requirements are similar and modest performance loss is acceptable.

While absolute performance is critical, clinical deployment should also consider inference cost and latency, model and API stability, data governance and privacy, output reliability, and system integration.

Fine-tuned transformers provide efficient, predictable output formatting and can be deployed using fixed model versions and local inference on consumer-grade hardware without relying on cloud APIs.

By contrast, proprietary, closed-weight frontier LLMs such as GPT-5.3 Chat offer strong performance but incur API costs and raise cloud-related privacy concerns. Open-weight LLMs, if deployed locally, offer greater control over data and model versions but require substantial infrastructure. However, the evaluated open-weight LLMs underperformed GPT-5.3 Chat in our study. LLM outputs may also require strict validation (eg, 2-pass DP text alignment to ensure accurate character-position mapping to the original unfiltered reports for keyword highlighting) before integration into existing systems.

Based on performance requirements, inference volume, and available annotation and computational resources, we provide practical recommendations for the following deployment scenarios.

For high-volume negation filtering requiring high performance, human-labeled transformers achieved the highest overall performance, with substantially lower recurring inference costs and latency than direct frontier LLM inference. On 1200 test reports, ALBERT required 57.0 seconds and approximately US $0.01, compared with 35 minutes 11 seconds and approximately US $21.8 for GPT-5.3 Chat.

Thus, with sufficient annotation resources, they remain the most practical option for large-scale institutional deployment.

Direct few-shot frontier LLM inference offers a viable alternative for high-performance, low-volume applications, such as filtering CT and MRI impression sections for subsequent retrieval.

With only 10-shot prompting and no task-specific fine-tuning, GPT-5.3 Chat achieved performance close to that of transformers trained on thousands of expert-labeled reports (ΔF₁-score 0.009, 95% CI 0.000-0.017; large-sample regime). Its higher per-report cost may be acceptable at low volumes because it avoids the need to construct large task-specific datasets, fine-tune models, and maintain dedicated models, provided that other institutional requirements are met.

However, our results still favor transformers for high-volume institutional deployment, although declining LLM inference prices over time [38] may broaden the range of workloads for which direct frontier LLM inference becomes economically feasible.

When both expert-annotation resources and recurring inference budgets are limited, LLM-based KD offers an affordable alternative with a modest performance trade-off. LLM-generated pseudo-labels enable transformer fine-tuning, shifting LLM expenditure from recurring inference to largely one-time annotation costs while retaining the low cost and latency of transformer inference.

Our results showed no consistent performance differences among students trained with different LLM teachers when all teachers achieved strong performance (F₁-score ≥0.950; Table 4). Thus, lower-cost LLMs may serve as practical alternatives to more expensive frontier proprietary models for pseudo-label generation, provided that downstream transformer performance is carefully validated. This strategy may particularly benefit institutions with limited expert-annotation and inference resources.

Given the strong few-shot performance of frontier LLMs, a hybrid workflow that combines LLM-generated outputs with clinician review can balance scalability and oversight. With high baseline accuracy in LLM outputs, experts need to only revise a limited number of negation spans rather than to annotate everything from scratch, reducing workload while maintaining label quality. Clear guidelines and structured reviews remain critical to avoid anchoring bias and error propagation from LLM predictions.

We found that integrating a string-alignment algorithm into the distillation pipeline can create word-level correspondence between LLM outputs and source texts, extending model predictions to individual words. Future studies can apply this framework to sequence-labeling or information-extraction tasks, such as named entity recognition [40], where each word’s position may convey important context.

Our analysis of error patterns in models using distilled labels, compared with human-supervised models, identifies 2 targets for improving LLM-based KD.

First, future studies could test causal mechanisms of downstream performance loss by replacing erroneous LLM-generated labels for selected instances, such as those involving symbolic or telegraphic-style negation, with human labels while holding all other labels and experimental factors constant. Performance recovery would support a causal contribution of these errors rather than a mere association.

Second, studies could apply distillation methods with richer teacher information, such as replacing hard pseudo-labels with confidence-aware or probabilistic teacher signals to determine whether richer supervision improves transfer. Such interventions could further distinguish limitations in the teacher’s predictive performance from those introduced by the distillation method.

Our study did not employ parameter-efficient fine-tuning (PEFT). PEFT methods, such as LoRA (low-rank adaptation) [41], may reduce the computational and memory requirements of model adaptation, particularly for larger models such as LLMs. However, full fine-tuning of our transformers required modest computational resources, making data annotation a greater resource burden than model training. As PEFT reduces model adaptation costs rather than labeled-data requirements, it complements rather than replaces our KD framework.

Regarding model performance, PEFT may offer additional gains beyond few-shot inference when sufficient task-specific data are available [42]. However, given the high performance of 10-shot GPT-5.3 Chat (F₁-score 0.974), we did not further fine-tune the LLMs using either full-parameter fine-tuning or PEFT. Future studies could assess the benefits of PEFT for larger models.