Manually creating clinical notes has always been a time-consuming and exhausting task for health care providers. As health care systems grow increasingly complex and large-scale, the need for faster and more accurate documentation methods has become more pressing. Transformer architecture has brought significant advancements to various natural language processing (NLP) tasks, including text summarization—a fundamental task in NLP. These advancements, driven by transformer-based large language models (LLMs) and the availability of large-scale datasets, have the potential to revolutionize health care systems. An NLP-powered system can analyze doctor-patient conversations, identify relevant clinical facts, structure the information, and generate coherent medical reports. By automating the generation of clinical notes, such systems provide timely insights and support to medical professionals during patient interactions. Real-time information retrieval ensures clinicians have immediate access to relevant medical data and patient histories, which can aid in making critical decisions. This, in turn, leads to more accurate diagnoses, personalized treatment strategies, and improved patient outcomes. Beyond individual patient interactions, the ability to analyze data at scale enables medical facilities to make data-driven decisions that enhance overall treatment quality, optimize resource use, and improve patient satisfaction. These innovations promise to streamline health care processes and elevate the standard of care.

The challenge lies in ensuring these automated notes are precise. Any mistakes or hallucinations in medical facts could have serious consequences. Summarizing clinical dialogues is tricky, but we tackled this by using decoder-only transformer models, which we found outperform traditional sequence-to-sequence models (such as Flan-T5-Large) on metrics such as Recall-Oriented Understudy for Gisting Evaluation (ROUGE) and bidirectional encoder representations from transformers (BERT) scores. Using the Medical Training Summarization Dialog (MTS-Dialog) dataset [1] containing 1700 doctor-patient conversations and their summaries, we explored models such as Mistral [2] and Llama [3], with Llama3 [4] emerging as the top performer, even beating the best results from the 2023 MEDIQA-Chat challenge [5].

Manual note-taking can be time-consuming, diverting health care providers’ attention from patient care. On average, physicians dedicate approximately 52 to 102 minutes each day documenting clinical notes based on their patient interactions [6]. Automatic clinical note generation can be a solution to this problem. It can reduce the burden of paperwork on health care providers and improve the accuracy of the medical records. This allows doctors to focus more on patient care rather than on paperwork. During the COVID-19 pandemic, face-to-face doctor visits were restricted. For that reason, health care systems experienced over a 100% surge in virtual urgent care appointments and more than a 4000% rise in virtual nonurgent care visits [7]. Automatic clinical note generation can help us to overcome this kind of situation. Clinical notes can vary widely in terms of content, format, and quality. Automated systems can help to standardize documentation, improve data quality, and facilitate analysis. Nevertheless, automated systems can extract valuable insights from clinical notes, enabling data-driven decision-making and improving patient care.

Our research set out to fine-tune the LLMs to craft high-quality clinical notes that make a real difference. We aimed to find a model that balances speed and precision, fine-tuning it on MTS-Dialog to adapt to the dataset’s unique demands. Our ambition was to go beyond the current leader, Flan-T5-Large, and establish a new benchmark for automated documentation. To measure our success, we compared our model’s ROUGE and BERT scores against Flan-T5-Large, assessing its ability to summarize accurately while preserving every essential detail.

The study by Ben Abacha et al [1] introduced the MTS-Dialog dataset, pairing simulated doctor-patient dialogues with clinical notes. They tested transformer models such as Bidirectional and Auto-Regressive Transformers (BART) [8] and Pegasus [9], finding BART, especially when prefinetuned and guided, produced the most accurate notes. However, issues such as hallucinations and missing key details persisted, highlighting both the promise of automation in health care documentation and the need for better factual accuracy. The study by Ozler and Bethard [10] explored LLMs for summarizing medical dialogues in the MEDIQA-Chat 2023 competition. Using models such as Clinical-T5 and Roberta-base on Medical Information Mart for Intensive Care datasets, they hit a peak accuracy of 72.3%. Limited hardware and dataset size posed challenges, but their work shows LLMs’ potential for medical documentation, with room for improvement through advanced models and techniques. The study by Sharma et al [11] tackled the same competition, fine-tuning BART-large on datasets such as Medical Information Mart for Intensive Care-IV-Note and introducing an N-pass strategy to summarize long dialogues. Data augmentation (DA) with synthetic dialogues boosted results, though hallucinations remained an issue. Their research pushes clinical NLP forward, suggesting future integration of medical knowledge. The study by Wang et al [12] used ChatGPT and BioMedLM in a doctor-patient loop system for MEDIQA-Chat 2023, excelling in dialogue generation and note summarization. While effective, gaps in medical knowledge and handling lengthy conversations need work, pointing to future refinements in segmentation and expertise. The study by Suri et al [13] evaluated transformer models such as Bio-Bart and DialogLED for the same challenge. DialogLED-Large outperformed GPT-3, offering a cost-effective alternative despite limited training data. They recommend DA to improve reliability. The study by Tang et al [14] fine-tuned BART and CONFIT while leveraging GPT-4 for MEDIQA-Chat 2023. GPT-4’s natural outputs impressed human experts, though privacy concerns with external application programming interfaces surfaced. Their findings underscore LLMs’ role in streamlining clinical notes. The study by Srivastava [15] tested local-sparse-global BART ensemble methods, finding that section-wise models outperformed single approaches for chart note summaries. Multilayer techniques and PubMed fine-tuning fell short, suggesting specialization as a key focus moving forward. The study by Milintsevich and Agarwal [16] fine-tuned FLAN-T5 and LongT5, using multitask learning to cut hallucinations in clinical notes. While effective, adding clinical named entity recognition tags unexpectedly hurt quality, indicating a need for better augmentation strategies. The study by Zhang et al [17] combined an SVM classifier with GPT-3 for summarization, with GPT-3.5 outperforming a fine-tuned Curie model. Their hybrid approach hints at the power of blending traditional and modern techniques. Finally, the study by Mathur et al [18] used GPT-4 with in-context examples, topping MEDIQA 2023 rankings. Despite concise outputs, brevity and privacy risks remain challenges, marking a step forward in applying LLMs to health care. In another research, the study by Heilmeyer et al [19] demonstrated that hospitals can use smaller, open-source LLMs running on their own computers to handle sensitive documentation, rather than relying on commercial systems. They found that a model specifically optimized for the local language (German) performed exceptionally well, generating medical reports that doctors rated as usable 93% of the time. The study by Savage et al [20] showed that while basic fine-tuning is enough for simple medical checklist tasks, using the more advanced direct preference optimization method allows LLMs to handle complex clinical reasoning and triage by teaching them to distinguish between high-quality and poor responses.

This study uses the MTS-Dialog dataset [1], a publicly available collection of simulated doctor-patient conversations. As the dataset contains no protected health information or real patient interactions, this research does not classify as human participants research. A qualified physician participated in this study to evaluate the clinical accuracy of the generated notes. This involvement was strictly in a professional capacity as a domain expert to assess text quality, rather than as a research participant. No personal data were collected from the evaluator.

In most cases, the graphics processing unit (GPU) hardly has enough memory to fine-tune any decoder-only LLM. To overcome this problem, we used the parameter-efficient fine-tuning (PEFT) technique proposed in the study by Houlsby et al [22]. The authors in the paper proposed a parameter-efficient transfer learning technique for NLP that introduces small trainable adapter modules into every layer of a pretrained model with frozen original model weights. The technique minimizes trainable parameters, leading to efficient and scalable fine-tuning across various tasks. We also used 8-bit quantization, where the original pretrained weights of the model are quantized to 8-bit and kept fixed during fine-tuning. This method is known as quantized low-rank adapter (QLoRA) [23].

The results of this study demonstrate that decoder-only transformer models, specifically Llama3-8B and Meditron-7B, achieve SOTA performance among open-source, parameter-efficient, fine-tuned models in generating clinical notes from doctor-patient dialogues. Our findings indicate that PEFT with 8-bit quantization (QLoRA) allows these large models to perform effectively under hardware constraints while maintaining high medical accuracy. The results clearly show that DA did not enhance the performance of either model; both the Mistral and Llama variants performed better without it. A similar phenomenon has been noted and explained in a previous research paper [28]. Notably, an expert physician’s evaluation confirmed that these models produce clinically coherent and accurate summaries, with Llama3-8B and Mistral-7B leading in overall clinical quality.

Our research establishes a new benchmark by surpassing the previous leader, Flan-T5-Large, in automatic evaluation metrics such as ROUGE and BERTScore. Unlike some entries in the MEDIQA-Chat 2023 competition that relied on DA to boost performance, our experiments showed that DA actually decreased performance for Mistral and Llama variants, aligning with findings in other specialized medical NLP research.

Table 5 shows the training and inference times for different models. Training time is measured in seconds/epoch. All fine-tuned models are inferred with a Tesla T4 GPU. From Table 5 it is visible that the inference time for Mistral-7B is better than Llama-3-8B, but it is worse than Meditron-7B. However, for real-world applications, it needs to be improved further. Training time for both models is quite long.

All the fine-tuned models had some gender biases. As the “MTS-Dialog” dataset is a short doctor-patient conversation dataset, sometimes it is quite difficult to understand the patient’s gender from the conversation. In these situations, all of the fine-tuned models mostly assume the patient’s gender as male. In a few cases, the model predicts the gender as female, while in the annotation, the patient is identified as male. In some cases in the dataset, the gender pronoun is used incorrectly. We assume that most of the model’s pretraining dataset could be biased, which is causing this type of problem. In some cases in the dataset, the gender pronoun is used incorrectly. To overcome this, we use the following new prompt during inference:

However, even after using this prompt, there was still no significant improvement. This gender-aware prompt was evaluated qualitatively but is not reflected in these baseline quantitative results.

Another major limitation of our quantitative evaluation is the reliance on a single expert physician, which introduces potential evaluator bias and precludes the calculation of interrater reliability metrics such as Cohen κ. Future studies should use multiple independent evaluators to mitigate individual bias.

The amount of data in the medical domain are very limited. It is also very hard to get access to this type of data because of the physician-patient privilege. Most of the patients are not comfortable sharing their private data. It is essential to develop more data, which will make it easier to create an automatic clinical note generation system. The dataset used in this research is short conversations between doctors and patients. A more and longer real-world doctor-patient conversation corpus is needed in the future to improve the quality of clinical note generation.

Domain-specific pretrained decoder-only LLMs have improved the domain-specific task a lot in recent years. A pretrained model such as Code Llama is one example in the coding domain [29]. Developing a decoder-only model that is pretrained on a large medical corpus might help to create clinical note generation tasks.

In diverse health care settings, doctors and patients may converse in different languages. A cross-lingual summary can help bridge this language gap by automatically summarizing conversations in one language and translating the summary into another, making it accessible to a wider range of health care providers. Creating a cross-lingual clinical note generator could greatly impact this domain.

In this research, text data from doctor-patient conversations are used. Generating clinical notes directly from the spoken interactions between doctors and patients could provide a more accurate and practical solution.

The maximum sequence length was set to 512 tokens due to hardware memory constraints. This truncation represents a methodological limitation that may have affected the completeness of the generated notes for longer dialogues. Future implementations should consider sliding window approaches to handle lengthy conversations without information loss.

It is also shown that just updating the prompt while doing inference is not a solution to this problem. The hypothesis of the authors of this paper suggests these steps to overcome the problem. The wrongly addressed gender in the dataset should be corrected manually in the reference note, and the patient’s gender information should be included in the conversation. After that, the prompt should be updated by including the information to detect the correct gender of the patient and use this prompt to fine-tune the model. By doing all these tasks, it may be possible to solve the problem.

In this research, we experimented with various decoder-only transformer architectures to fine-tune models for generating clinical notes by summarizing conversations between doctors and patients. The results demonstrate that decoder-only models such as Llama3 and Mistral outperform classical encoder-decoder models such as Flan-T5 and Pegasus in summarizing medical discussions. Larger models tend to deliver better results compared to smaller ones. By fine-tuning the Llama-3-8B and Meditron-7B models, SOTA performance among open-source, parameter-efficient models was achieved in terms of ROUGE-1 score and BERTScore. However, none of the models underwent full fine-tuning. Instead, PEFT methods were applied across the board. The study highlights the importance of domain-specific pretraining and high-quality annotated datasets. While our methods show promising results, there are still challenges to overcome, such as hardware limitations, gender bias, and the lack of diverse medical datasets. Ultimately, automating clinical documentation through decoder-only LLMs has the potential to enhance health care efficiency. However, given the unresolved gender bias observed in this study, these models are currently best suited as assistive tools in human-in-the-loop workflows rather than autonomous systems, requiring further research into bias mitigation before they can ensure fully accurate patient records.