Radiology reports consist of expert observations of the radiologist based on the Chest X-ray images of the patient. These reports consist of free-text and unstructured information in the form of long paragraphs. The extraction of labels from unstructured radiology reports is the task of radiology report labeling and it provides us structured information which can be used for many downstream tasks such as medical report generation and natural language explanation generation. It also enables training of large-scale medical imaging models [1]. Previous works for labeling of radiology reports involve use of complicated feature engineering of medical domain knowledge [2]. and Bidirectional Encoder Representations from Transformers (BERT) based approaches [3]. Transformers have also demonstrated success in radiology report labeling [4,5]. However, all these methods have limitations which hinders their adoption in the clinical setting. In case of methods which use complex feature engineering, these methods have leveraged manual annotation to shift the burden from feature engineering, requiring considerable time and expertise. Furthermore, these methods do not take advantage of existing feature-engineered labelers, which are state-of-the-art on many medical tasks. On the other hand for methods using BERT based models, the models are limited by the inherent limitations of BERT models such as their noncausal nature and limited context length. BERT-based models, despite their effectiveness in text classification tasks, have two key architectural limitations that constrain their performance in radiology report labeling. First, BERT’s bidirectional nature focuses on context aggregation but lacks the ability to model causal relationships in sequential data. This noncausal nature can hinder its ability to fully capture the hierarchical and temporally dependent structure of radiology reports, where findings are often sequentially described. Second, BERT’s limited input context length (typically 512 tokens) prevents it from effectively processing the long and detailed narratives commonly found in radiology reports. As a result, crucial information in extended texts may be truncated, leading to incomplete or suboptimal labeling. These limitations reduce the adaptability of BERT-based methods to real-world radiology settings, where comprehensive understanding of the entire report is often required.

Large language models (LLMs) such as Qwen address these challenges by offering extended context lengths (several thousand tokens), allowing the model to process full radiology reports without truncation. In addition, their ability to incorporate causal reasoning and handle instruction-based tasks makes them particularly suitable for medical labeling tasks, where nuanced and ambiguous language is prevalent. Although BERT based methods have shown increased abilities in the classification and other natural language tasks, their architecture poses a hindrance to their use.

Smit et al [3] introduced a combination of existing radiology report labelers and expert annotations to achieve highly accurate automated radiology report labeling . Their approach consists of a bio-medically pretrained BERT model [6,7], which is trained on the outputs of an existing labeler. They call their resulting model CheXbert. While CheXbert has generated considerable results, it has been unable to capture the full diversity, complexity and the ambiguous nature of natural language in the radiology reports. Their BERT-based solution while providing remarkable performance on the task of labeling is limited in its context length and noncausal nature. Which means for longer radiology reports it fails to provide a solution.

In our work we propose an LLM based radiology report labeler. LLMs have proven to be successful in their natural language generation capabilities. These models also have longer context lengths which makes them highly suitable for natural language generation tasks. Furthermore, these LLMs are adept at following instructions and given proper instructions these LLMs can be made good labelers for the radiology reports. Our LLM-based model inherently provides ease of use in other LLM-based solutions for the medical domain enabling clinical automation.

Our generative pretrained transformer (GPT)-based LLM beats the BERT-based CheXbert model on many pathologies and with a far bigger context length can handle long reports as compared with CheXbert. Our model outperforms the previous labelers [8] for many pathologies on an external dataset, MIMIC-CXR [9]. Our method of training medical report labelers opens room for other labels and longer textual input which makes it broadly useful for natural language processing tasks within the medical domain.

Many natural language processing systems have been developed to extract structured labels from unstructured free-text radiology reports [2,10-16]. Mostly, these methods rely heavily on feature engineering and include strict vocabulary and grammatical rules to find and classify radiological reports. NegEx [17], a popular rule-based method, uses simple regular expressions for detecting negation of findings and is often used in combination with the Unified Medical Language System [18]. NegBio [19], an extension to NegEx, uses universal dependencies and subgraph matching for pattern definition and graph traversal search. It includes uncertainty detection in addition to negation detection for multiple pathologies in radiology reports, and is used to label the ChestX-Ray14 dataset [1]. The CheXpert labeler [8] improves upon NegBio on chest x-ray report classification by more controlled extraction and an improved Natural Language Processing framework and rules for uncertainty and negation extraction. The CheXpert labeler has been applied to generate labels for the CheXpert dataset and MIMIC-CXR [9], which are among the largest chest x-ray datasets publicly available. We use the MIMIC-CXR dataset to train our LLM-based framework and report our findings on a subset of the test set of MIMIC-CXR which has been labeled by expert radiologists.

Previous approaches have also been trained using radiology reports annotated by expert radiologists [20]. In these approaches, training data is limited by radiologist time and expertise. Chen et al [21] trained convolutional neural networks with Global Vectors for Word Representation [22] on 1000 radiologist-labeled reports for classification of pulmonary embolism in chest computed tomography scan reports and improved upon the previous rule-based peFinder [23,24]. trained both recurrent and convolutional networks in combination with attention mechanisms on 27,593 expert-annotated radiology reports. Transformer-based models have also been applied to the task of radiology report labeling [4], trained BERT [6] and XLNet-based [25] classifiers on 3856 radiologist labeled reports to detect normal and abnormal labels. Wood et al [5] proposed ALARM, an MRI head report classifier on head MRI data using BioBERT model [26] trained on 1500 radiologist-labeled reports. They demonstrate improvement over previous fixed embedding and word2vec-based [27] models [28] CheXbert labeler [3]. Also proposed a BERT based model which is trained on expert annotated radiology reports and achieves state of the art results for radiology report labeling. However, their method has limitations which include restriction to the context length of 512 which is the limitation of BERT based models.

Recent advancements in the application of LLMs across various domains, including medical informatics, have demonstrated their versatility and efficacy. Models such as GPT-3 and GPT-4 have been used for diverse tasks, including automated clinical note generation, question answering in health care, and medical coding, showcasing their ability to handle complex and domain-specific language tasks. In radiology, LLMs have been explored for summarizing imaging findings, generating patient-friendly explanations, and aiding in clinical decision-making, highlighting their potential beyond classification tasks. Furthermore, instruction-tuned LLMs, such as ChatGPT and specialized variants like BioGPT, have been shown to adapt effectively to biomedical domains, opening avenues for tasks such as multimodal data interpretation and real-time clinical assistance. These advancements emphasize the need for further exploration of LLMs’ contextual understanding and adaptability, particularly in radiology report labeling and other biomedical text processing tasks.

In our work we propose an LLM based solution to solve the task of biomedical text labeling. We not only propose an alternative to BERT based models which achieves better scores on certain labels and has far bigger context length than BERT based models but our approach can also be applied to other biomedical text labeling.

Radiology report labeling is a critical task involving the extraction of information on the presence or absence of specific thoracic pathologies, such as consolidation or edema, from free-text radiology reports. This process enables the transformation of unstructured diagnostic text into structured data, facilitating clinical decision-making, research and the development of predictive models. In this task, a labeler processes the free-text radiology report as input and assigns 1 of 4 classes, blank, positive, negative, or uncertain, to each of 14 predefined observations, reflecting the certainty level for each prediction. A “positive” label indicates the presence of a pathology, while “negative” denotes its absence, and “uncertain” is used when the report is ambiguous about the condition. The “blank” class is assigned when no relevant information is available for an observation. By converting radiology reports into labeled data, this approach supports streamlined access to essential diagnostic insights and provides a valuable structured dataset that can be used to train and validate machine learning algorithms in medical imaging.

In radiology, there exist 2 large datasets of chest x-rays, CheXpert [8] (consisting of 224,316 images), and MIMIC-CXR [9] (consisting of 377,110 images). Both datasets have corresponding radiology reports that have been labeled for the same set of 14 observations using the CheXpert labeler [8] from the Impression section, or other parts of the radiology report. Furthermore, a subset of both datasets also contain manual annotations by expert radiologists. On CheXpert, a total of 1000 reports (CheXpert manual set) were reviewed by 2 board certified radiologists with disagreement resolution through consensus. On MIMIC-CXR, a total of 687 reports (MIMIC-CXR test set) were reviewed by 2 board certified radiologists and manually labeled for the same 14 medical pathology labels as in CheXpert. However, the radiology reports for the Chexpert dataset have not been made public. Due to nonavailability of radiology reports for the Chexpert dataset, we used the MIMIC-CXR test set for evaluation.

LLMs are built upon stacked decoder layers from the transformer architecture, often referred to as “auto-regressive models” because of their causal structure. This auto-regressive nature enables these models to predict each token sequentially, relying only on the preceding tokens as context. During training, LLMs learn the task of next-token prediction, where they must accurately anticipate the subsequent token in a sequence given the previous tokens as input. This prediction task is essentially a binary classification problem, where the model assesses whether its predicted token matches the correct ground truth token. The accuracy of each prediction is evaluated by calculating a cross-entropy loss, which measures the difference between the model’s output and the correct token. This loss is then back-propagated through the model to update its weights, refining its ability to generate contextually accurate and coherent responses over time.

LLMs are pretrained on extensive amounts of diverse text data from vast resources available on the internet. This extensive pretraining enables the models to capture complex patterns and detailed knowledge present in language, making them highly effective at understanding and generating natural language. Their exceptional performance in natural language generation, coupled with their ability to handle longer text inputs, motivates their application in various specialized domains, such as radiology report labeling.

In our work, we specifically use the Qwen model [29], particularly the Qwen1.5‐0.5B variant. This model demonstrates significant capabilities in various natural language generation tasks relative to its moderate size, making it both efficient and powerful. One of the Qwen model’s key advantages is its extensive context length of 32,000 tokens, meaning it can handle considerably larger text inputs as context compared with BERT-based models, which are typically limited to a 512-token context. This increased context capacity allows the Qwen model to process lengthy radiology reports or extended medical dialogues without truncating important information.

The enhanced context length in LLMs is particularly beneficial in clinical settings, where maintaining continuity in patient information, such as previous history or ongoing conversations, is essential. As LLM-based frameworks are increasingly adopted in health care, often in the form of biomedical chatbots and other automated systems, the capacity to retain extensive context is critical. Our model addresses this need, providing a solution that integrates seamlessly with existing LLM-driven tools in clinical environments. This ensures that radiology report labeling and other clinical tasks benefit from both accuracy and the ability to preserve a comprehensive, context-aware understanding of patient data.

To fully use the capabilities of our pretrained LLM, we fine-tune it using a specialized instruction dataset. This dataset is constructed from radiology reports paired with corresponding pathology labels, each with an associated certainty level from the MIMIC-CXR dataset. The goal of this instruction tuning is to guide the pretrained LLM to understand the relationships within radiology reports, enabling it to accurately identify pathologies and assign certainty levels. By providing targeted instructions, we aim to refine the LLM’s ability to interpret the clinical language and nuanced patterns within radiology reports, thereby enhancing its performance in radiology report labeling.

The structure of the instruction dataset is designed to facilitate clear guidance for the LLM. As illustrated in Figure 1, each data instance consists of an instruction in the form of a prompt to the LLM, an input value (the radiology report), and output values (the pathology labels along with their certainty levels). This structured approach helps the LLM understand both the format and task requirements, allowing it to generate accurate labels from free-text radiology reports.

To prepare the data for model processing, each instance in the instruction dataset is tokenized using the LLM’s tokenizer, converting text into a sequence of tokens suitable for input. The tokenized data is then fed into the LLM, which generates predictions based on its learned representations. Following this, we calculate the cross-entropy loss, which measures the difference between the model’s predicted outputs and the ground truth labels. This loss value indicates how closely the model’s predictions align with the actual labels. By back-propagating the loss, we adjust the LLM’s weights which gradually enhances its accuracy and reliability in producing pathology labels with certainty levels.

Through this fine-tuning process, our LLM becomes adept at associating the textual features of radiology reports with relevant pathology labels, allowing it to accurately and efficiently label clinical data. This approach ensures that the model is optimized specifically for radiology report labeling, enabling it to perform well even on complex clinical information.

We trained our LLM, which has a total of 463,987,712 parameters, using an instruction dataset. During the training process, we maintained a batch size of 2, which allows for effective gradient estimation while minimizing memory usage. This choice of batch size is particularly beneficial given the substantial size of our model and the complexity of the task. The training was conducted over 5 epochs, providing sufficient iterations for the model to learn the intricate relationships between the radiology reports and their corresponding pathology labels.

To optimize the training process, we used the ADAM optimizer [30] which facilitated faster convergence and improved performance. In addition, we implemented a gradient accumulation strategy with a value of 2. This approach effectively simulates a larger batch size by accumulating gradients over two iterations before performing a weight update, allowing us to maximize the use of available GPU memory while still benefiting from the stability of a larger batch size.

The learning rate was set at 1 × 10⁻⁴, a value that strikes a balance between training speed and stability. An appropriate learning rate is crucial in preventing the model from oscillating around the optimal solution, ensuring gradual and consistent improvements in performance. For the instruction tuning of our LLM, we used the SFTTrainer from the TRL library [31]. This specialized trainer is designed for supervised fine-tuning, providing an efficient and effective framework for adapting pretrained models to specific tasks. It offers a range of features that streamline the training process, including automated handling of training loops, logging, and monitoring of performance metrics. We leveraged 4 NVIDIA RTX A6000 GPUs. This powerful hardware setup enables parallel processing, significantly reducing training time while accommodating the memory requirements of our large model.

To fully use the capabilities of our pretrained LLM, we fine-tune it using a specialized instruction dataset.

In this study, we evaluate the performance of our model, CheXbert and the CheXpert labeler, across a suite of retrieval tasks designed to assess their capabilities in clinical information extraction. Specifically, we focus on 3 main retrieval tasks, namely positive extraction, negative extraction, and uncertainty extraction. For each task, we designate the relevant class as the “positive” class for classification purposes, meaning, for example, that the “negative” class is treated as the positive class in the negative extraction task, while other classes (such as positive or uncertain) are treated as negatives. This approach allows us to directly measure the model’s ability to distinguish between the specified class and all others, thereby assessing its precision and recall within clinically relevant categories.

We compute the weighted average of the F1 scores for each of the 14 clinical findings or observations present in the CheXpert dataset across these tasks. F1 score is a harmonic mean of precision and recall which provides a balanced measure of a model’s classification performance, particularly in settings where data is imbalanced. We calculate a weighted average of the F1 scores for each of the 14 observations across these tasks. By this we can mitigate the impact of class imbalance and obtain an F1 metric that accurately reflects the model’s performance across both common and rare observations in the dataset.

This weighted metric, referred to as weighted-F1, is denoted simply as F1 in our results. Finally, we calculate and report the average F1 score across all 14 clinical observations, offering insight into the model’s general extraction capability. This average F1 score serves as a key performance metric, allowing for a direct comparison between CheXbert and the CheXpert labeler on clinical information extraction tasks relevant to medical imaging applications in radiology .

Currently, our work is limited to 14 pathology labels prevalent in radiology. However, our method can be extended to other medical fields and can accommodate many more pathology labels and can provide a reliable labeling solution. The lack of long form radiology text limits the demonstration of the longer contextual abilities of our model. In future with the advent of long form bio-medical text in the form of patient-doctor conversations, our model can provide better solutions as opposed to previous models. Furthermore, there is a need of more publicly available radiologist annotated datasets for additional evaluation of trained models. Furthermore, like other LLM-based approaches, our model is susceptible to biases present in the training data, which may inadvertently propagate into its predictions. This is particularly important in the medical domain, where biases can have significant ethical and clinical implications. Additional mechanisms for bias detection and mitigation are necessary to ensure fair and equitable outcomes across diverse patient populations.

The computational demands of LLMs are another limitation, as training and inference require substantial computational resources. This may limit the accessibility and scalability of the approach, especially in resource-constrained clinical settings. Exploring techniques such as model distillation or low-rank adaptation could help reduce these requirements while maintaining performance.

Interpretability remains a challenge with LLMs due to their complex architecture. The lack of transparency in how decisions are made can hinder trust and adoption in clinical workflows, where clear justification of predictions is often required. Future work should explore ways to improve the explainability of LLM predictions, such as attention visualization or feature attribution methods tailored to medical tasks.

While our method demonstrates promising results in radiology, further validation is needed in real-world clinical settings and across diverse medical contexts. The scalability of our approach to other modalities, such as MRI or CT reports, and its adaptability to different healthcare systems must also be explored to ensure generalizability.

In our work, we propose a novel solution using LLMs for the challenging task of radiology report labeling. Our approach leverages the advanced capabilities of LLMs to accurately interpret and label various pathologies within radiology reports, which are crucial for supporting clinical decision-making and improving patient care. We demonstrate that LLMs achieve superior performance over previous BERT-based models, especially in accurately identifying specific pathology labels. This is a significant advancement, as certain pathologies that were previously challenging to label accurately can now be identified with greater precision using our LLM-based approach. In addition, our model achieves competitive scores across a broad range of pathology labels, confirming its robustness and versatility in handling complex medical language.

One of the key improvements introduced by our model is its ability to process longer radiology reports, overcoming the context length limitations often encountered in previous models. This capability is essential in clinical settings where comprehensive radiology reports often contain detailed descriptions spanning multiple paragraphs, which traditional models struggle to handle effectively. By accommodating extended context, our model ensures that no critical information is overlooked, thereby enhancing labeling accuracy and reliability in real-world applications.

Furthermore, the instruction-tuning methodology allows our approach to be extended to other biomedical text labeling tasks beyond radiology. This generalizability of our method makes it highly applicable across a wide range of biomedical domains, opening doors for enhanced automation in various clinical documentation and reporting tasks.

As LLM-based solutions are increasingly adopted in clinical settings, our model is designed to integrate seamlessly into existing workflows, promoting efficiency, and facilitating clinical automation. The potential for real-time, accurate labeling provided by our model not only reduces the manual workload on health care professionals but also contributes to more timely and precise diagnostic insights, ultimately benefiting patient outcomes.

Future research should explore integrating LLMs with other medical technologies, such as multimodal imaging systems that combine radiology reports with visual data from x-rays, CT scans, or magnetic resonance images. This integration could enable a more comprehensive analysis and further enhance the clinical utility of LLM-based models. In addition, extending the application of LLMs to different medical imaging modalities and nonradiology domains, such as pathology or cardiology, presents a promising avenue for expanding their impact across health care.

Developing scalable and efficient LLM solutions tailored to specific clinical needs, including resource-constrained environments, is another critical area for further exploration. This includes investigating lightweight alternatives or model compression techniques to reduce computational demands while maintaining performance.

Finally, the clinical adoption of LLMs will require continued efforts in improving model interpretability and building trust among health care professionals. Future work should focus on developing user-friendly interfaces and explainability mechanisms to facilitate the seamless integration of LLMs into routine clinical workflows. By addressing these challenges, LLM-based models like ours have the potential to revolutionize clinical practice, enabling more accurate, efficient, and scalable solutions for medical documentation, decision-making, and patient care.