Task-Specific Transformer-Based Language Models in Health Care: Scoping Review

Introduction

Background

Transformer models have revolutionized natural language processing (NLP) with their exceptional state-of-the-art performance in various applications such as conversation, translation, text classification, and text generation. A transformer model is a type of deep learning model designed to process and generate sequences of data, such as text. The key innovation of transformer models is the self-attention mechanism, which allows the model to weigh the importance of different words in the input sequence, regardless of their position. Self-attention allows the model to focus on different parts of an input sequence simultaneously, rather than processing the sequence in a fixed order. This mechanism enables the model to capture complex patterns and relationships within the context more effectively than previous models, which is particularly useful for understanding and generating natural language. These models hold significant promise for the health care sector, addressing clinical challenges and unlocking new opportunities in medical informatics (eg, disease prediction, clinical decision support, and patient interaction).

Since the introduction of the transformer model by Google [1] in 2017, it has become the foundation for various pretrained language models (PLMs). PLMs are transformer models that have been initially trained on a large text corpus before being fine-tuned for specific tasks. This pretraining allows the models to leverage vast amounts of unstructured data to improve their performance in various NLP tasks. Two of the most widely used PLM architectures in medical research are Generative Pre-trained Transformer (GPT) and Bidirectional Encoder Representations from Transformers (BERT). GPT is designed to generate coherent text based on a given input, making it useful for tasks like dialogue generation [2]. BERT, on the other hand, is designed to understand the context of words in a sentence from both directions, making it highly effective for tasks like question answering and text classification [3]. Transformer-based language models have revolutionized the field of NLP and continued to advance the state-of-the-art in NLP with their impressive performance.

Despite the success of transformer-based language models in many domains, there is a significant gap in comprehensive reviews specifically focused on their application in the health care domain. In health care, transformer-based language models have been used for crucial tasks such as disease prediction, decision-making, and image analysis [4]. With the abundance of free text sources, such as medical documentation in free text, including social media, electronic medical records (EMRs), physician-patient conversations, and online encyclopedias, more significant challenges to language models are needed. The application of NLP in health care is not without controversy, particularly concerning data privacy, ethical implications, and the integration of artificial intelligence (AI) systems into clinical practices. Debates continue about the extent to which AI can replace human judgment, the transparency of AI decision-making processes, and the potential biases in AI models trained on unbalanced datasets. By addressing these concerns, our paper contributes to the timely and critical discourse on the responsible deployment of transformer-based language models in health care, emphasizing the need for transparency, fairness, and ethical considerations in AI development.

Objective

The objective of this paper is to provide a comprehensive scoping review of task-specific transformer-based language models in health care. By focusing on models pretrained on medical corpora, we aim to address the gap in existing literature where detailed surveys specifically tailored to health care applications are lacking. We seek to highlight the strengths, limitations, and potential of these models, offering valuable insights for future research and practical applications in medical informatics.

Related Work

While many review studies of NLP have been conducted in the medical field [5-13], on transformer-based language models [14-20], and in health-related domains [21-25], comprehensive surveys and broader and up-to-date transformer-based language models in health care are lacking, leaving a gap in understanding their full potential and limitations. Pandey et al [5] introduced RedBERT, a model focusing on topic discovery and deep sentiment classification of COVID-19 online discussions, demonstrating the application of NLP in understanding public health concerns. Iroju and Olaleke [6] conducted a systematic review of NLP applications, identifying key areas where NLP can enhance clinical decision-making and patient care. Similarly, Locke et al [7] provided a comprehensive overview of NLP in medicine, emphasizing the potential of NLP technologies in transforming medical practice. Adyashreem et al [8] surveyed various NLP techniques in the biomedical field, shedding light on how these techniques can be applied to biomedical text for improved information extraction and analysis. Wang et al [9] reviewed the application of NLP in clinical medicine, highlighting the advancements and challenges in integrating NLP with clinical workflows.

Khanbhai et al [11] applied NLP and machine learning techniques to patient experience feedback, providing insights into patient satisfaction and areas for improvement in health care services. Casey et al [12] focused on NLP applications in radiology reports, identifying how NLP can streamline the interpretation and reporting of radiological findings. Zhou et al [13] discussed the broader applications of NLP for smart health care, envisioning a future where NLP-driven systems enhance patient care and operational efficiency.

In the realm of transformer-based language models, Zhang et al [14] surveyed their applications in bioinformatics, highlighting how these models have advanced the analysis of biological data. Yang [15] and Lin et al [16] explored the progress and applications of transformer models in Korean and general NLP tasks, respectively, highlighting their growing importance and versatility. Chitty-Venkata et al [17] reviewed neural architecture search for transformers, underscoring the potential of these models in optimizing NLP tasks. Gillioz et al [18] provided an overview of transformer-based models for various NLP tasks, illustrating their adaptability and efficiency. Han et al [19] focused on multimodal pretrained models, emphasizing their capability to handle diverse data types, including text, image, and audio. Greco et al [20] and Albalawi et al [21] discussed transformer models’ applications in mental health and Arabic social media, respectively, highlighting their potential in understanding and addressing specific health-related issues. Kalyan et al [22] and Shamshad et al [23] provided comprehensive surveys on biomedical PLMs and their applications in medical imaging, respectively, showcasing the transformative impact of transformers in these fields.

Our review categorizes these models into 6 key tasks: dialogue generation, question answering, summarization, text classification, sentiment analysis, and named entity recognition (NER). Ultimately, advancements in transformer-based language models hold the promise of significantly transforming health care delivery and improving patient outcomes. By enabling more accurate disease prediction, enhancing clinical decision support, and facilitating better patient-provider communication, these models can lead to more efficient, effective, and personalized health care. Our review underscores the broader implications of these technologies, advocating for continued research and development to harness their full potential in revolutionizing medical informatics and patient care.

Methods

Information Source and Search Strategy

We followed the Cochrane scoping review protocol to conduct and map the available literature in an efficient and systematic approach. This method involves defining the research question, identifying relevant studies, selecting studies based on predefined criteria, charting data, and summarizing the results to clarify key concepts and identify research gaps [24].

Our research team (mainly HNC and TJJ) conducted a comprehensive literature review for identifying studies in the field that met the inclusion and exclusion criteria. The screening and selection of papers were conducted by 2 independent reviewers (HNC and TJJ). Initially, titles and abstracts were screened to identify relevant studies. Full texts of potentially eligible studies were then reviewed to ensure they met the inclusion criteria. Disagreements between reviewers were resolved through discussion and consensus, with a third reviewer (YHK) consulted if necessary. Our literature search was conducted across several scientific databases, including Google Scholar and PubMed, which were selected for their comprehensive coverage of relevant journals and peer-reviewed studies in the medical and academic fields. We covered publications from January 01, 2017, to September 30, 2024, and used specific combinations of keywords and Boolean operators, such as “transformer-based AND language models AND medical domain,” “health care AND language models,” “NLP AND medicine AND survey,” and “GPT AND BERT AND health care.” Data extraction involved summarizing key findings, model names, and training datasets. The extracted data were cross-verified by both reviewers to ensure accuracy and consistency. Any discrepancies were resolved through discussion.

We included studies that involved transformer-derived models applied to medical tasks, were published in peer-reviewed journals, and were written in English. The exclusion criteria involved studies focusing solely on non-text data (eg, audio, image, and video) or those not meeting the inclusion requirements. The selection of tasks (dialogue generation, question answering, summarization, text classification, sentiment analysis, and NER) was based on their critical role in advancing health care applications of transformer models. The specific process is illustrated in Figure 1, with details of each stage of filtering from the initial identification of articles to the final selection. The inclusion criteria were rigorously applied at each step, beginning with the screening of titles and abstracts, followed by a full-text review, and culminating in the inclusion of studies that met all predefined criteria. This methodical approach allowed us to compile a comprehensive and focused set of articles for our scoping review, ensuring that our findings are both robust and reliable.

These tasks cover a wide range of functionalities essential for improving clinical workflows, enhancing patient interactions, and facilitating efficient information retrieval and analysis, making them vital for the advancement of transformer-based language models in the medical domain. Languages and model types were chosen to represent a diverse range of medical contexts and applications.

‎

In this section, studies that have used language models in health care applications were examined. Based on the literature review, Table 1 provides a comprehensive list of transformer-based models applied in the medical domain, comparing each task based on the authors, model name, training dataset, PLM model, key metric, score, and purpose or findings of the study. These English-written PLMs in the health care domain were categorized into 6 distinct tasks, namely dialogue generation, question answering, summarization, text classification, sentiment analysis, and NER. The articles within each task are listed in no sequential order. In Figure 2, the evolution timeline of transformer-based language models provides an overview of significant models that have been developed for use in medicine. It illustrates key milestones and the deployment criteria used to guide the inclusion of studies in our review. This historical context provides a foundation for understanding the methodological choices made in our scoping review. This visual representation highlights the emergence of models over time and their increasing significance in health care applications. We provide insights into the progress made in this field and anticipate future advancements by tracking the development of these models.

‎

Results

Selected Studies

A total of 75 models were identified through our comprehensive review. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart is presented in Figure 3. The PRISMA checklist is presented in Multimedia Appendix 1. These papers encompass various research areas related to transformer-based models and their applications in the medical domain. The selection of these papers was based on predefined inclusion criteria, ensuring the relevance of each study to the scope of our review.

‎

Applications of Language Models in Health Care: Task-Specific

Dialogue Generation

Conversation generation generates responses to a given dialogue. GPT models, including DialoGPT and DialogBERT, can effectively generate human-like dialogues based on large corpora and contextualized representations of text [25,102-105]. In the medical domain, conversation generation focuses on developing conversations related to medical information [106]. Chatbots in health care can be classified into 6 types: screening and diagnosis, treatment, monitoring, support, workflow efficiency, and health promotion. These tasks involve aiding patient consultation, acting as a physician’s decision support system, collaborating with interdisciplinary research, and providing care instructions and medical education [107,108].

The key models are MEP, BioBART, MedPIR, MEDCOD, Transformer-DST, MeDiaBERT, ChatDoctor, and SciFive.

Research efforts on conversation generation in medicine have also incorporated knowledge graphs. MKA-BERT-GPT was the first scalable work to integrate a medical knowledge graph mechanism into a large pretrained model. Meanwhile, MedPIR proposed a recall-enhanced generator framework by using a knowledge-aware dialogue graph encoder to strengthen the relationship between the user input and the response via past conversation information [28,30]. They achieved an F1 score of 82% and a bilingual evaluation understudy (BLEU) score of 21.5. Varshney et al [26] proposed the Masked Entity Dialogue (MED) model to train smaller corpora texts, addressing a 10% improvement in entity prediction accuracy for the problem of local embeddings in entity embeddings by incorporating conversation history into triples in the graphs, resulting in an automatic prediction of the medical entities model.

On the other hand, MEDCOD [31] used the GPT pretrained model to integrate emotive and empathetic aspects into the output sentences, which further imitates a human physician–like feature to better communicate with patients. The Transformer-DST [34] model addresses dialogue state tracking, optimizing state operation prediction, and value generation with high accuracy by suggesting to ask the DST model to consider the whole dialogue and the previous state. Moreover, MeDiaBERT [35], using a hierarchical approach, achieves 63.8% accuracy in multiple choice medical queries by building a transformer encoder contained within another in a hierarchical manner.

BioBART [27], a BART-based model, used patient descriptions and conversation histories as input for the model to autoregressively generate replies to user inputs. The model outperformed the BART model by 1.71 on Rouge-2 with a BLEU score of 4.45, and pretraining on PubMed abstracts supported the model’s performance. OPAL [29] and -w terms+AL models also used BART for pretraining. OPAL’s proposed method involves 2 phases: pretraining on large-scale contextual texts with structured information extracted using an information-extracting tool and fine-tuning the pretrained model on task-oriented dialogues. The results showed a significant performance boost, overcoming the problems created by annotated data with large structured dialogue data. Recently, the -w terms+AL model proposed a framework for improving dialogue generation by incorporating domain-specific terminology through an automatic terminology annotation framework using a self-attention mechanism [33].

While other models are based on BERT, GPT, or BART, ChatDoctor [32], SciFive [36], and PMC-LLaMA [37] use LLaMA or T5 PLMs. To improve accuracy and provide informed advice in medical consultations, ChatDoctor used Meta’s open-source LLaMA [109], which was fine-tuned using real-world patient-physician conversations and autonomous knowledge retrieval capabilities, achieving 91.25% accuracy. SciFive, a Text-To-Text Transfer Transformer–based model, was pretrained on large biomedical corpora, indicating its significant potential for learning large and extended outputs. The SciFive model was trained using a maximum likelihood objective with “teacher forcing” [110] for multi-task learning by leveraging task-specific tokens in the input sequence. Both models outperformed previous baseline methods.

More recently, the HuatuoGPT model [38], specifically tailored for the Chinese medical domain, provided state-of-the-art results in medical consultation tasks.

Question Answering

The question-answering task involves answering questions posed by users based on the texts in documents. It aims to generate an accurate response that directly answers the question input, contributing to clinical decision-making, medical education, and patient communication. Allowing physicians and researchers to obtain valuable answers quickly from electronic health records (EHRs) and various medical literature will effectively reduce the time and effort required when the procedure is done manually. While the dialogue generation and question-answering tasks both involve providing answers, the former focuses on generating responses within a conversation, whereas the latter focuses on developing specific answers to user questions.

The key models are BioBERT, BioGPT, BioMegatron, Med-BERT, UmlsBERT, SMedBERT, and BioVAE.

BERT-based language models have become increasingly popular in biomedical text mining as they can understand the context and generate accurate predictions. BioBERT [39], the first domain-specific BERT-derived transformer language model for biomedical text mining applications, achieved 89% accuracy on the MedQA dataset and outperformed BERT in medical text applications. A BioMegatron model [41], based on Megatron-LM [111], was also experimented on a question-answering task, and it was found that the domain and task-specific language model affected the overall performance rather than the model size. Shin et al [41] found that model size is not closely related to the performance rate, but rather the domain and task-specific language model affects the overall performance. Med-BERT [42], another BERT-inspired model, improved the prediction accuracy by 20% in disease prediction studies by pretraining on EHR datasets.

More recently, researchers have built BERT-based models for specific domains and tasks [112]. UmlsBERT [44] first built a semantic embedding linking concepts with the words in the UMLS Metathesaurus and proposed multi-label loss function–masked modeling. SMedBERT [45] also presented a similar approach with the knowledge semantic representation but structured the neighboring entities to learn heterogeneous information. UmlsBERT and SMedBERT enhanced performance, with F1 scores of 84% and 86%, respectively. Similarly, LinkBERT and BioLinkBERT [43] incorporated ontological knowledge to better understand a linking system between entities in the corpus. LinkBERT used a multi-task learning framework on several related tasks simultaneously to extract relations between entities in the corpus more effectively. ExKidneyBERT, CaBERTnet, and MMBERT extracted more precise answers from individual departmental reports [46,47,49].

On the other hand, BioVAE [48] used the OPTIMUS framework pretrained with SciBERT [113] and GPT-2 [114,115] and outperformed the baseline models on biomedical text mining. To address the issues on linguistic disparity, SPBERTQA [50] proposed a 2-stage multilingual language model pretrained on the SBERT model [116] to reply to user questions using multiple negative ranking losses with Bert multilingual 25.

However, previous studies using the BERT structure are a better fit for understanding the context, rather than generating texts. To this end, BioMedLM, a GPT architecture model, was built mainly for biomedical question-answering tasks [117] in recent studies of question-answering benchmarks and achieved 50% accuracy on summarizations of the patient’s quest even in real situations with fewer data. BioGPT [40] applied a 2-step fine-tuning method to remove the noise in data and achieved 6.0% improved results compared with BioLinkBERT in the medical domain for question-answering tasks.

Recent studies have introduced significant advancements, such as BioMEDGPT [51], the first multimodal GPT for aligning biological data with human language, achieving 76.1% accuracy. Clinical Camel [52], using LLaMA-2, demonstrated superior performance with 5-shot accuracy ranging from 47.0% to 74.3%, outperforming GPT-3.5. MedAlpaca [53] focused on privacy and medical certifications, attaining 21.1%-24.1% accuracy. MedPaLM-2 [54] reached 67.6% accuracy through instruction prompt tuning, and MEDITRON [55] achieved 79.8% accuracy, marking a 6% improvement upon existing models and setting a new benchmark.

Summarization

For many years, the medical field has suffered from the challenge of finding efficient and rapid access to understanding the fast-growing and immensely increasing amount of data formation. The key to timely and efficient clinical workflow is providing automatic summarization in clinical text. Summarization in health care is an important technique in NLP as it automatically summarizes the medical contexts into a concise summary of text. Summarization can be applied to medical records, literature, clinical trial reports, and other types of medical texts that aim to provide clinical providers with quick access to relevant information, without the need to skim through lengthy documents. Overall summarization can aid clinicians with decision-making through effective and prompt communication during the physician-patient meeting, as well as knowledge discovery for medical research [57].

The key models are BioBERTSum, AlphaBERT, ClinicalBertSum, ChestXRayBERT, RadBERT, LF2BERT, and DAMEN.

To alleviate the problems of biomedical literature summarization, which can have difficulties in learning sentence and document-level features, Du et al [57] proposed the first PLM for medical extractive summarization application called BioBERTSum. BioBERTSum captures a domain-aware token and sentence-level context by using a sentence position embedding mechanism that inserts structural information into a vector representation. It achieved a ROUGE-L score of 0.68, outperforming standard BERT models. AlphaBERT [60] proposed a diagnostic summary extractive model using a character-level token to reduce the model size and achieved a ROUGE-L score of 0.693, reducing the burden of physicians in the emergency department regarding reading complex discharge notes of patients.

To better use clinical notes, ClinicalBertSum [118] used the ClinicalBERT, SciBERT, and BertSum models during the fine-tuning and summarization process to automatically extract summaries from clinical abstracts. Similarly, ChestXRayBERT used BERT to perform an automatic abstractive summarization on radiology reports [62], with ROUGE-1 scores of 0.70 and 0.73, respectively. RadBERT [56], which was fine-tuned for radiology report summarization, achieved 10% fewer annotated sentences during the training, demonstrating the benefit of domain-specific pretraining to increase the overall performance.

LF2BERT [63] applied a Longformer neural network and BERT in an encoder-decoder framework to process longer sequence inputs and performed better than human summarization, according to doctors’ evaluations. DAMEN [59] used BERT together with BART to discriminate important topic-related sentences in summarization, outperforming previous methods to summarize multiple medical literature via the token probability distribution method. The proposed probabilistic method selected only related significant chunks of information and then provided the probabilities of the tokens within the chunk, rather than the sentence level, to effectively reduce redundancy. Moreover, to overcome the long sequence issue, Li et al [58] comparably proposed Clinical-Longformer and Clinical-Big Bird pretrained on the Longformer [58] and Big Bird [119] models, respectively. Both proposed models used sparse attention mechanisms and linear level sequence lengths to mitigate memory consumption, thus increasing long-term dependency to train extensive clinical notes.

Recent development has also introduced MEDVOC [64], which uses GPT architecture to improve the adaptation of vocabulary in medical texts. By efficiently reducing fine-tuning time, MEDVOC achieves competitive performance, with ROUGE scores of 51.49, 47.54, and 19.51 across different datasets, such as PubMed, BioASQ, and EBM, respectively.

Text Classification

The medical text classification task categorizes medical text datasets into predefined categories based on the content and context within the text [120]. Disease classification, medical image classification, drug classification, and sentiment analysis are some of the standard text classification applications in health care.

The key models are jpCR+jpW, BioMed-RoBERTa, ClinicalBERT, Mask-BERT, KG-MTT-BERT, EduDistilBERT, PathologyBERT, KD distilledBERT, MatSciBERT, and Bioformer.

Wada et al [120] proposed a BERT model, jpCR+jpW, that uses a classification method. The method pretrains the medical BERT model once following the up-sampling step of domain-specific word amplification. This is done to achieve better performance on a smaller medical corpus. Similarly, BioMed-RoBERTa [70] used the RoBERTA model and applied a domain and task-adaptive pretraining strategy with a simple data selection approach for domain-specific classification. By pretraining on domain-specific and unlabeled data, the model achieved 87% accuracy. ClinicalBERT [67] represents clinical notes effectively, with a word similarity accuracy of 90% to generate visualized and interpretable embeddings for capturing semantic associations between clinical texts.

Yogarajan et al [121] suggested applying multi-labels (eg, using more than 300 labels for longer documents) to enhance the performance of medical classification tasks. Furthermore, to solve the imbalance class problems, Rodrawangpai et al [122] demonstrated a framework of adding normalization layers and dropout to BERT-based models, which improved the classification performance by 4% on data that included imbalance target labels. Similar efforts have been made by Nguyen [50] to address label-abandoning problems in medical abstract classification. The author proposed that a BERT model with label attention in the fine-tuning process raised the F1 score by 0.3 and supported the explainability of the prediction results. Learning is difficult with insufficient labeled data in a low-resource experiment setting. To alleviate this problem, Mask-BERT [71] proposed a framework for few-shot learning, where the mask is applied to the input text and enables the gathering of more definitive tokens. Masked learning leads to filtered results on anchored samples from the data being used for representation, increasing the robustness of the output features.

Several language-specific models have been developed, including RuBioBERT and RuBioRoBERTa [123] for Russian text, BERTurk [124] for Turkish text, BioGottBERT for German text [88], and a Spanish text model [125], demonstrating the applicability of BERT-based models beyond English. Moreover, various disease-specific classification models have been developed. For example, PathologyBERT [126] is a pretrained masked language model used for classifying the severity of breast cancer diagnoses, raising the importance of applying domain-specific tokenization. KD distilled_BERT [127] is a response-embedded knowledge distillation framework that used pretrained BERT for depression classification and achieved a high accuracy of 97%. MatSciBERT [68] presents a biomedical domain-specific classification model on abstracts of the literature with binary classification application. The model extracted the context of the embeddings alongside the topic and had 2.75% higher accuracy than SciBERT.

To overcome the over-fitting and dimensionality problems for extracting numerous features in the text classification task, AFKF [128] proposed a fusion block with Kalman filters onto features of EMRs. This led to a 20% increase in accuracy compared with previous models. Likewise, to classify the features in EMRs, BERT-MSA [129] showed that a multilayered self-attention mechanism improved accuracy in obtaining relevant features. EduDistilBERT [130] demonstrated that adapting a smaller BERT model with limited parameter usage increases overall performance by 95% while reducing the computation cost. Another BERT-based fusion approach by Al‑Garadi et al [131] explored architecture to fuse BERT, ALBERT, and RoBERTa model probabilities using a naive Bayes classifier, achieving an F1 score of 0.67 in classifying medication abuse texts.

Recently, the Bioformer [69] model demonstrated a 60% reduced model size and a 2- to 3-fold increase in performance speed. The model used a whole-word masking approach with 15% masking, which provided contextual information. However, KG-MTT-BERT [72] raised a question on limitations for multi-type clinical text classification. Concatenating numerous texts may be more efficient in developing relevant contextual information, and using only BERT may misplace crucial details. Therefore, the model extended the BERT model with a knowledge graph during fine-tuning, demonstrating effective handling in classifying patients into diagnosis-related groups.

Although the models were pretrained on the BERT model, Gao et al [132] showed that BERT-structured models did not gain better accuracy on clinical classification tasks, such as classifying discharge summaries or pathology reports, compared with nontransformer language models. Gao et al asserted that, in addition to the knowledge obtained through the entities, grammar patterns should also play a role in the model’s mechanism. Furthermore, beyond the applications mentioned above, text classification can be used with other tasks. For example, Wang et al [133] applied a question-answering task along with the classification task by using the BERT model to classify texts in question inputs from patient inquiries regarding their symptoms.

Recent advancements in text classification models include TransformEHR [73], which uses BERT and longitudinal EHR data for clinical disease prediction, achieving area under the receiver operating characteristic curve and area under the precision-recall curve scores of 81.95 and 78.64, respectively. MeDa-BERT [74] tailored embeddings for Danish medical text, with accuracy ranging from 86.7% to 97.1%. SCHOLARBERT [75] leveraged public resource-driven datasets for scientific NLP, obtaining an F1 score of 85.49%. MediGPT [76] improved medical text classification tasks, with accuracy and F1 scores of 90.0% and 88.7%, respectively, showing a 22.3% performance gain over traditional methods.

Sentiment Analysis

The sentiment analysis task captures and identifies expressions and opinions [134] in medical contexts, including clinical notes, social media posts related to medicine, or patient feedback. For instance, sentiment analysis can capture the perception of people expressed in social media during the COVID-19 outbreak [135-137]. Emotions, such as positive, neutral, and negative sentiments, expressed by the public dominated during the pandemic [138]. Additionally, multi-label sentiment classification proved that the BERT model provided better performance compared with the LSTM model [139]. Moreover, the opinions of patients and physicians can be used to describe the symptoms and diagnosis to facilitate the decision-making process and support the decisions in clinical patterns [77]. The primary goal of sentiment analysis in health care is to provide insights into patient experiences, such as attitudes toward health care services and overall medical experience satisfaction. It not only assists patients but also supports clinicians to identify any underlying issues in patient care.

The key models are MentalBERT, MeentalRoBERTa, SINA-BERT, SGTN, RedBERT, T-BERT, AKI-BERT, TopicBERT, TweetBERT, and BelabBERT.

In mental health, patients’ written texts have become a valuable source for supporting hypotheses and providing insights into the emotions expressed by patients [85,140]. While more research using PLMs needs to be conducted in this field, MentalBERT, MentalRoBERTa [77], PsychBERT [140], and belabBERT [85] applied mental health texts and achieved sentiment classification accuracies of 75%, 86%, and 90%, respectively. Additionally, transformer language models have been studied for sentiment analysis in languages other than English. Some language models are being developed to accommodate the unique structure and characteristics of different languages.

To achieve an effective model adaptation, researchers have explored HeBERT and HebEMO [141] for Hebrew, AraBERT and MARBERT [142] for Arabic, SINA-BERT [78] for Persian, and Fine-tuned BERT [143] for Chinese. However, few studies have conducted disease-specific sentiment analysis. RedBERT [80] involved a sentiment model for COVID-19, where BERT was used for classifying sentiments of Reddit comments to grasp insights into the pandemic. Mao et al [82] proposed AKI-BERT, where the model was developed to support the early prediction of acute kidney injury.

Social media data are often used as a source for medical sentiment analysis as they are more informal and conversational, making them useful for modeling the nuances of language models. The COVID-TWITTER-BERT model [144], which was pretrained on Twitter messages regarding COVID-19, showed improved performance on COVID-19–related datasets. In particular, TweetBERT [84] exhibited improved performance on COVID-19–related and biomedical datasets. TwitterBERT was evaluated on 12 different biomedical datasets and outperformed previous BERT models, such as SciBERT [113] and BERT [145].

Comparably, TopicBERT [83], a memory-efficient BERT model, fine-tuned and enhanced sentimental analysis performance, and a complementary topic framework was applied to improve its performance. Beyond the proposed frameworks presented above, AlBadani et al [79] proposed a graph transformer model, SGTN, which used BERT to pretrain node embeddings and aggregated neighboring information to efficiently learn sentiments. It showed 5% improvement over baseline models.

Named Entity Recognition

The NER task identifies the named entities in unstructured text data. In health care, NER is used to automatically extract and define relevant medical entities, including diseases, medications, procedures, and other clinical concepts, from medical texts in research papers or EMRs [146,147]. The common applications of NER in medicine are as follows: (1) identify and analyze medical entities and relationships in medical literature to support biomedical findings [148]; (2) extract patient data, such as diagnosis, medication, laboratory results, and physical measurements, from EMRs to improve the decision-making for clinicians and the overall care [149]; and (3) extract and categorize data from medical claims and hospital admission and discharge data to improve health care management and resource allocation [150,151].

The key models are Bio+ClinicalBERT, Med-BERT, G-BERT, BioALBERT, GatorTron, ELECTRAMed, CamemBERT, BioGottBERT, Ra-RC, and RG-FLAT-CRF.

Numerous language models have been developed to implement EMR or EHR data for NER tasks in the clinical field. Bio+Clinical BERT [61] achieved superior results in clinical texts, with an F1 score of 83%. While the ClinicalBERT and Clinical BioBERT models were trained on EHRs, the Bio+Clinical BERT model did not perform well on deidentification text. Other models, such as G-BERT, also used EHR data to propose a language model that combined graph neural networks and BERT for representing medication information and predicting drug recommendations [87]. MedGPT [96] effectively processed noises by organizing medical text in multi-step procedures. In the first stage of the proposed model, unstructured data were converted into a standardized ontology using NER+L. Then, GPT was used for forecasting diagnosis events.

Moreover, studies attempted to tackle problems regarding representing and learning long medical entities [94,152]. Liu et al [152] proposed Med-BERT, using a Span-FLAT method for longer medical entities, and it achieved an F1 score of 84%. By contrast, the BIBC model built by Yang et al [94] captured both local and global sequence features to efficiently solve long text input issues. Additionally, models were trained on an extensive collection of biomedical texts to overcome the limited amount of training data. For example, BioALBERT and GatorTron attempted to develop a large medical language model [65,93]. BioALBERT used vocabulary specifically tailored to the biomedical domain and applied the ALBERT structure. On the other hand, GatorTron used the byte pair encoding algorithm and was pretrained on the GPT model to scale up the language model up to 8.9 billion parameters, showing 9.6% accuracy improvement. Furthermore, the datasets in the medical domain face the challenge of not only limited training data but also low-quality labeled training data. Therefore, multi-task learning was presented by Khan et al [92], and the slot tagging problem was approached with MT-BioNER, a multi-task transformer-based model that enhanced memory performance and time efficiency in slot tagging, with 10% better performance than single-task models.

While recent studies have heavily relied on BERT-based structures, transformer models used other PLMs for improving NER tasks. ELECTRAMed [91] proposed an ELECTRA-based model for the biomedical domain, which reduced the sequence length and training phases. Additionally, many models have focused on multilingualism, including the CamemBERT, BioGottBERT, Ra-RC, and RG-FLAT-CRF models [95,153,154], focusing on efficiently learning features in languages other than English, such as French, German, and Chinese.

More recent studies include HeartBERT [98], which emphasizes department-specific models, focusing on cardiology and achieving 74% accuracy. UMLS-KGI-BERT [99] introduced graph-based learning for clinical text extraction, with a precision of 85.05%. CardioBERTpt [100], which is specialized in Portuguese cardiology terms, improved NER performance, with an FL-score of 83%. Finally, TocBERT [101], which is fine-tuned on the MIMIC-III dataset, outperformed rule-based methods for segmenting discharge summaries, achieving an F1 score of 84.6%.

Discussion

Principal Findings

This study examined previous studies on transformer-based language models in the medical domain. We reviewed a total of 75 recently studied models that aligned with our inclusion criteria. The initial step of the method involves categorizing the models based on the tasks they perform, such as dialogue generation, question answering, summarization, text classification, sentiment analysis, and NER. Then, each study is analyzed based on the key findings, frameworks, pretraining models used, and model names. Finally, the limitations of each task application are discussed. The use of transformer-derived language models in medicine has shown numerous advantages, such as high accuracy, language comprehension, automated diagnosis, adaptability, and efficiency. However, these models also face several challenges, including the lack of standardization, the need for domain-specific knowledge, limited annotated training data, safety concerns, interoperability and interpretability issues, integration complexities, ethical considerations, and evaluation issues. In this discussion section, we explore key limitations and future directions of models in terms of each task and the generalizability of the explored models across different health care settings and population considerations. Model challenges, their potential solutions, and the future of natural language models in the medical domain will be discussed.

Specific Task-Based Challenges and Future Directions

Dialogue Generation

Dialogue generation models like DialoGPT and ChatDoctor face challenges such as handling the complexity and specificity of medical terminology, ensuring data privacy, and providing accurate, contextually relevant, and empathetic responses. Privacy and security issues are critical since these models deal with sensitive patient information. The risk of privacy, chances of errors, ethical constraints, and security issues remain to be addressed. The challenge comes from the specificity and complexity of medical terminology, although the medical dialogue system certainly should provide only accurate and informative knowledge tailored to the level of expertise of the end user. Therefore, human experts need to conduct regular risk and security audits.

The following suggestions are made to further improve medical dialogue research. First, continuous learning and training of the dialogue system are necessary to incorporate up-to-date knowledge for users. Additionally, language translation could be integrated into the dialogues to enable universal access to data and promote a more profound exchange of insights without language barriers. Moreover, chatbots [107] should be integrated in a real medical setting to reduce medical costs and physician burdens. Proper and accurate usage of the dialogue system may assist patients in navigating through the vast amount of freely available online data, finding correct information, and avoiding falsified or unsolved answers. Lastly, automated data augmentation techniques can be used to create unbiased dialogues. These suggestions can lead to further advancements in medical dialogue research, leading to more efficient communication between patients and medical professionals.

Question Answering

Medical question-answering systems like BioBERT and UmlsBERT struggle with the complexity of medical terminology for nonexperts in the medical field. Patients who are experiencing an illness may find it difficult to filter and search for relevant information. These models need to handle diverse linguistic data and adapt regional variations in medical practices. One approach to addressing these limitations is to integrate multilingual models to handle questions in various languages. Another approach is the incorporation of region-specific medical data to improve model generalizability and accuracy. Further, enhancing the ability to integrate summarization tasks on top of the question-answering system may provide comprehensive responses. However, such a multi-task system requires several human experts to evaluate the provided answers in order to judge the task performance accurately.

Summarization

Medical summarization is a crucial application in language model tasks to facilitate the hospital’s process and significantly reduce the workload and burnout of clinicians. However, challenges emerge due to the complexity of health care terminologies and the need for expert knowledge to comprehend them. The ability to achieve concise and faithful summaries is critical for avoiding physician burnout and patient dissatisfaction. Models like BioBERTSum and ClinicalBERTSum face challenges in learning sentence and document-level features, handling complex medical terminologies, and ensuring summaries are concise and accurate. The risk of physician burnout due to extensive documentation can be mitigated by effective summarization. Future work can focus on developing a system of human expert assessments to validate the summarization quality. Additionally, combining extractive methods and abstractive summarization methods is suggested. A fine-tuned summarization model for a particular task should consider tense information and personal information. We recommend building an ensemble method to improve pretraining and fine-tuning datasets for summarization effectiveness. Medical summarization is a crucial application in language model tasks to facilitate the hospital’s process and significantly reduce the workload and burnout of clinicians.

Text Classification

Improving the accuracy and effectiveness of classification tasks poses several challenges and limitations that need to be addressed. Models, such as BioMed-RoBERTa and ClinicalBERT, need to address issues related to class imbalance, limited annotated data, and the complexity of medical terminologies. Limited training data, for instance, can be addressed by collaborating with different institutions to gather various information options in vocabulary usage and text structure, and high-quality annotated data can thus be developed. Ambiguity, variation, concept drift, data privacy, language complexity, and class imbalance can be addressed by employing domain-specific approaches, and pretraining language models can be leveraged on similar datasets. Domain-specific approaches can resolve ambiguity issues and achieve active learning to reduce the reliance on large volumes of labeled data. These strategies will facilitate better model performance results.

Sentiment Analysis

Despite previous research results, medical sentiment analysis remains a challenging task due to personalized information required to accurately measure meaning and interpret emotions in context. MentalBERT and RedBERT, for instance, need to accurately interpret emotions in medical contexts, handle personalized information, and manage the complexity of evaluating representations in the biomedical domain [155]. Organizing emotions in context requires sentiments, including sarcasm, emojis, and misspelled words, which create subjectivity, as noted by Brezulianu et al [156].

These limitations can be overcome by defining emotional polarity for annotations and integrating cultural, economic, and medical contexts into the model. Future research should consider using a domain-specific sentiment dataset, adapting the specific medical source (despite the lack of an available dataset, mostly from a single source), creating highly effective and defined labels in data, performing analysis based on the context, building both the explicit and implicit sentiment lexicon, and addressing the lack of a mental health–related sentiment lexicon. By addressing these challenges, future work can develop more accurate and effective medical sentiment analysis language models.

Named Entity Recognition

NER downstream tasks are imperative to address its limitations. The limited annotated data in medical text datasets is a major challenge due to the high cost and time involved in labeling, resulting in limited labeled data for model training. The Bio+ClinicalBERT and Med-BERT models face challenges in handling limited annotated data, normalizing various terminologies, and ensuring accurate entity extraction across different medical texts.

Therefore, we suggest collaborative effort among health care providers, biomedical researchers, and computer engineering experts to develop effective and robust NER models. Improving the annotation algorithms and creating extensive and accurately labeled medical text datasets can significantly enhance the performance. Moreover, standardized clinical entities can prevent ambiguity arising from abbreviations and context. The use of transfer learning techniques and domain PLMs can be beneficial in addressing the limited annotated data issue. Developing domain-specific dictionaries and ontologies can aid in improving the model performance.

Generalizability Challenges

Health care systems vary widely in their practices, protocols, and terminologies. For instance, a model trained on data from the United States may not perform optimally in a health care setting in Asia or Europe due to differences in clinical infrastructure and settings. The availability of resources, such as EHRs and technological infrastructure, can also differ between urban and rural settings, and between developed and underdeveloped countries. This variability can significantly affect the implementation and performance of the models.

Moreover, patients from different ethnic and cultural backgrounds may present symptoms differently and may have varying health behaviors, and models need to account for these varying characteristics to avoid biases and ensure equitable health care delivery. Moreover, multilingual populations pose a challenge for language models trained predominantly on English language data. The inclusion of diverse linguistic data during model training can mitigate this issue with a language-specific pretraining stage followed by a shared fine-tuning stage to improve the model’s applicability across different regions.

By incorporating diverse datasets during training, language models can support personalized medicine initiatives. This involves tailoring medical treatments to individual patient characteristics, leading to more effective and efficient care. Developing adaptable models that can be fine-tuned with local data ensures scalability across different health care settings, to address regional variations in medical practices and patient demographics.

Standardizing Medical Data for Improved Model Performance

The quality and consistency of medical data may vary across health care settings. Models trained on high-quality standardized data may not perform as well when applied to settings with less structured and lower quality data. The lack of standardized terminologies in medical texts, which encompass a vast array of terminologies from disease-specific to domain-specific language, is a notable challenge. Currently available datasets often have a restricted range of medical entities, posing difficulties in accurately extracting relevant entities.

To address this, we suggest creating standardized clinical entities. This would enable the normalization of different names or abbreviations to accurately normalize entities for standard medical terminology, thereby improving data consistency and model performance. Building standardized forms that are widely adopted and available in multiple languages will facilitate standardized medical learning. Additionally, developing domain-specific models has proven effective in enhancing model performance. For instance, an open-source package for detecting clinical entities from medical texts, which can recognize risk factors, medications, and diagnoses, can be developed to support this initiative.

The fuel of building and training language models is data. Collecting accurate information and precisely fabricating the data design during the preprocessing step is crucial. The challenges in creating such quality data for medical language models include a lack of key annotation and limited training data. Annotating medical text is time-consuming and costly, resulting in limited labeled data for training models. First, using diagnosis codes on weak supervision for training labels is suggested. Second, the pipeline should support the automated retrieval of datasets and multiple types of clinical entities to enable the preservation of annotation relationships across different languages. The automated retrieval of datasets and the development of speedy and supportive algorithms can aid in data integration and preprocessing. In addition to technical solutions, emphasizing the importance of multidisciplinary collaboration can significantly enhance the development and implementation of these models. By integrating expertise from various fields, we can overcome challenges, develop innovative solutions, and further advance the field of AI in health care. Collaborative efforts among data scientists, clinicians, bioinformaticians, and ethicists are crucial for building robust, reliable, and ethically sound models.

Ethical Considerations

Interoperability and cybersecurity pose significant challenges in medicine. EMRs and clinical decision support systems often have difficulty interacting with each other, leading to inefficiencies in patient care. To overcome these challenges, it is important to develop strategies focused on informed consent, safety, transparency, and algorithmic fairness for bias prevention. Ensuring that patients provide informed consent for the use of their data is critical. This involves informing patients about the use of their data, the benefits and risks, and their rights to withdraw consent at any time. This process upholds patient autonomy and enhances trust in AI systems.

Safety and transparency are fundamental to the ethical deployment of AI models in health care. Models handling sensitive patient data and providing clinical recommendations, such as BioBERTSum, must be rigorously validated and continuously monitored to detect and rectify errors promptly. Transparency can be achieved by making algorithms and decision-making processes understandable to users. This includes documenting how models are trained, the types of data used, and the underlying mechanisms of the algorithms.

Ensuring algorithmic fairness is crucial to prevent biases in AI models, which could lead to unequal treatment of patients. AI models trained on biased datasets can perpetuate existing disparities in health care. For example, models must include diverse and representative data to avoid underrepresentation of certain populations, ensuring fairness and accuracy across different groups such as ethnicity, cultural background, gender, and age. By addressing these ethical considerations within mitigated guidelines, we can ensure the reliability of transformer language models in medicine to improve overall health care while preserving fairness.

Evaluation Metrics

Furthermore, the rapid evolution of medical knowledge poses a challenge for language models to adapt and remain up to date with innovative discoveries. Ensuring the interpretability of language models is also crucial to address the trust issue and support the decision-making process. To evaluate medical language models, multiple metrics, including the F1 score, Biomedical Language Understanding Evaluation, Biomedical Language Understanding & Reasoning Benchmark (BLURB), and Chinese Biomedical Language Understanding Evaluation [157], should be used to overcome unbalanced performance issues.

Conclusion

We presented a comprehensive survey of task-specific transformer-derived models employed for diverse medical tasks, demonstrating their significant potential in the medical domain. Numerous studies have highlighted their capabilities in improving health outcomes, extending beyond disease prediction and medical classification studies. Our work clearly delineates the applications of transformer-based language models in various medical tasks such as dialogue generation, question answering, summarization, text classification, sentiment analysis, and NER. We identified innovative models and their unique contributions to the field. These findings distinguish our work from existing literature by providing a detailed, task-specific analysis of transformer-based models in health care.

Despite the promising advancements, several challenges must be addressed to develop effective models. These include standardization, limited annotated data, interoperability, and ethical considerations. To overcome these challenges, it is crucial to emphasize multidisciplinary collaboration. Future research should investigate transformer models that incorporate visual or audio data sources to provide a more comprehensive understanding of medical contexts.

Developing models that support patients’ experiences and assist health care practitioners in focusing solely on critical tasks by providing evidence-based recommendations and identifying potential diagnostic and treatment options can remarkably improve patient care. AI-driven tools rationalize administrative tasks, reduce paperwork, and improve workflow efficiency, eventually saving time for health care providers. Further, policymakers can leverage insights from transformer-based models to inform health care policies and allocate resources more effectively, ensuring equitable health care delivery.

This review solely focused on transformer language models that used text data. While the findings are promising, the applicability of these models may vary across different medical settings and populations. Our findings highlight the transformative potential of transformer-based language models in the medical field. By addressing the identified challenges and focusing on innovative research directions, the health care domain can advance significantly. We encourage researchers to build upon our work, address these challenges, and explore new frontiers in medical AI to improve patient care and clinical decision-making.