A Vision-Language–Guided Multimodal Fusion Network for Glottic Carcinoma Early Diagnosis: Model Development and Validation Study

Introduction

Glottic carcinoma (GC) is a common malignant tumor of the head and neck [1]. According to GLOBOCAN, China reported 29,500 new cases and 16,900 deaths from laryngeal cancer in 2022 [2], posing heavy burdens on health care systems. Approximately 60% of patients are diagnosed at an advanced stage [3], leading to significant impairment of vital physiological functions and compromising both physical and mental health. Early diagnosis of malignant tumors has been increasingly emphasized in clinical practice due to its potential to improve cure rates and organ function preservation [4]. Therefore, optimizing diagnostic methods for GC and enhancing early detection capability are urgent tasks for otolaryngologists.

Laryngoscopy is the primary diagnostic tool for GC [1], offering direct visualization of lesion shape, extent, and surface texture. When combined with narrow band imaging, it enhances early tumor detection by identifying neovascularization [5], making it a valuable tool for early diagnosis. However, vocal cord dysplasia (VCD), a precancerous condition situated between normal epithelium and squamous cell carcinoma, is characterized by a small lesion with clinical and laryngoscopic features similar to early GC [6]. Therefore, it is challenging for the human eye to distinguish between them. In addition, the lesion is often covered by “leukoplakia-like” substance, which interferes with the ability of narrow band imaging to reveal submucosal vasculature [7], increasing the risk of misdiagnosis. Laryngoscopy reports provide textual descriptions of lesion morphology observed dynamically during the examination, supplementing static images and assisting in diagnostic decision-making. Studies have demonstrated a correlation between morphological grading and malignancy risk [8-10], underscoring the diagnostic value of textual reports. Furthermore, reports authored by experienced clinicians serve as valuable references, facilitating more accurate diagnoses by less experienced clinicians. Histopathology examination remains the gold standard for diagnosis [1]. However, biopsy is invasive, painful, and carries procedural risks [11], impeding its widespread application in large-scale clinical screening. To address this issue, it is necessary to develop an efficient and noninvasive method to improve the diagnostic accuracy of early GC.

Recently, significant progress has been made in deep learning techniques for tackling real-world classification tasks in computer vision and natural language processing [12-14]. Many researchers have sought to apply these models to the detection of laryngeal cancer, yielding promising outcomes. However, most existing methods only use laryngoscopic images as input, including UC-DenseNet [15], MTANet [16], Dlgnet [17], RedFormer [18], and SAM-FNet [19]. Although these methods have demonstrated promising performance in laryngeal cancer detection and other tasks, they neglect certain latent information present in other modalities. Such information is typically inaccessible through unimodal approaches, thereby highlighting the advantages of multimodal methodologies [20,21].

Therefore, this study proposes a novel method named vision-language–guided multimodal fusion network (VLMF-Net) for early diagnosis of GC. Addressing the limitations of traditional single-modality diagnostic methods, our approach integrates the images and reports text of laryngoscopy to provide a more comprehensive representation of lesion characteristics. Using a pretrained vision transformer (ViT) [22] model for image feature extraction and a LlaMa3 [23,24] model fine-tuned for text processing, we achieve effective multimodal feature fusion. Compared to single-modality methods, our approach significantly improves the diagnostic accuracy and robustness, achieving an accuracy of 0.776 on real clinical datasets. This study highlights the potential of multi-modal fusion in clinical auxiliary diagnosis and provides new insights for reducing misdiagnosis rates and improving patient treatment outcomes.

Methods

Dataset

In our study, we constructed 2 datasets for model development and validation. First, we built an internal dataset for model training, validation, and testing. For the internal dataset, we collected data from 404 patients with glottic lesions at the First Affiliated Hospital of Sun Yat-sen University in Guangzhou, China. This dataset consists of 5799 professionally annotated image-text pairs, covering two types of lesions: VCD and GC. Each sample includes a laryngoscopic diagnostic report written by an experienced otolaryngologist and its corresponding laryngoscopic image.

In addition, to assess the model’s generalization ability and robustness, we constructed an external dataset. The external dataset was collected from the First People’s Hospital of Zhaoqing, consisting of data from 47 patients with glottic lesions between January 1, 2018, and August 31, 2024. This dataset includes 308 image-text pairs and strictly follows the principle of isolation from the training data, serving only for final performance evaluation. For detailed information on internal and external datasets, please refer to Figure 1 and Table 1.

‎

Model Architecture

Overview

Figure 2 illustrates the overall architecture of our proposed VLMF-Net model, which consists of 3 main modules: the laryngoscopic image encoder, the clinical report encoder, and the laryngeal feature fusion module. Specifically, the laryngoscopic image encoder is responsible for extracting visual feature representations from laryngoscopic images, while the clinical report encoder captures textual feature representations from the patient’s laryngoscopic examination findings. These features are then fused through the laryngeal feature fusion module. Finally, the fused features are passed through a fully connected layer to complete the classification task. The detailed implementation of each module in our proposed VLMF-Net model is as follows:

‎

Laryngoscopic Image Encoder

The laryngoscopic image encoder is an adapted version of a pretrained ViT [22] model. Considering that a single ViT model may introduce cross-modal discrepancies in multimodal tasks, potentially affecting model accuracy, we incorporated an additional Q-Former [25] module into the ViT model to mitigate the impact of modality differences and bridge the gap between image and text features. Q-Former is a trainable module based on Transformer that extracts and condenses visual features through alternating stacking of self-attention and cross-attention. Specifically, we first use ViT to extract features from the image, and then we input the extracted image features into a Q-Former module with frozen parameters to reduce modal differences.

Formally, let $f_{enc}(\cdot )$ denote the image encoder and $f_q(\cdot )$ denote the Q-Former module. Given an image $x_i$, first resize the image to 224×224×3, then the image feature $v_i$ is obtained as follows:

${\displaystyle {\displaystyle v_i=f_q\left(f_{\text{enc}}\left(x_i\cdot {\theta }_{\text{enc}}\right);{\theta }_q\right)}}$(1)

where ${\theta }_{enc}$ and ${\theta }_q$ are the weight parameters of the image encoder and the Q-Former module, the shape of $v_i$ is 1×1024. It is worth noting that in terms of the weights of the pretrained model, we adopt the transfer learning strategy: we directly use the pretrained weights of the BLIP-2 [25] model as the weight parameters of the laryngoscope image encoder, because a large number of studies have demonstrated the effectiveness of BLIP-2 in downstream tasks [26-28]. Such operations can ensure that the extracted image features and text features are in the same scale space, thus effectively avoiding catastrophic problems caused by modal differences.

Clinical Report Encoder

In the clinical report encoder module, we use LLaMA3 [23], an advanced large language model, as the text feature extractor to obtain textual feature representations from the patient’s laryngoscopic examination findings. This model is renowned for its exceptional ability to understand long-form text.

Formally, let $f_{LLaMA3}(\cdot )$ denote the clinical report encoder function. Given a clinical report $r_i,$ the process of obtaining the text feature $t_i$ can be formulated as follows:

${\displaystyle {\displaystyle t_i=f_{\text{LLaMa3}}\left(r_i;{\mathit{\theta }}_q\right)}}$(2)

where ${\theta }_q$represents the weight parameters of the clinical report encoder, the shape of $t_i$ is 1×4096. Notably, we adopt the same transfer learning strategy for the clinical report encoder’s weight parameters as we did for the laryngoscopic image encoder. Furthermore, considering that the text in our dataset is in Chinese, we use a fine-tuned LLaMA3 model trained on a Chinese dataset as the encoder’s weight source [24]. This approach effectively addresses the original LLaMA3 model’s limitations in Chinese language understanding.

Laryngeal Feature Fusion

In the 2 modules described above, we obtained image and text features. To effectively fuse these 2 modalities, we introduced the laryngeal feature fusion module, which aligns, maps, and integrates the features from both modalities. Specifically, we first use the vision projector $f_{vp}(\cdot )$ and the text projector $f_{tp}(\cdot )$ to map the image feature $v_i$ and text feature $t_i$ into a unified feature space:

${\displaystyle {\displaystyle v_i^{\prime }=f_{\text{vp}}\left(v_i;{\theta }_{\text{vp}}\right)}}$(3)

${\displaystyle {\displaystyle t_i=f_{\text{tp}}\left(t_i;{\theta }_{\text{tp}}\right)}}$(4)

Where $v_i^{`}$ and $t_i^{`}$ represent the mapped image and text features (their shapes are 1×512 and 1×2048), respectively, and ${\theta }_{vp}$ and ${\theta }_{tp}$ are the learnable parameters of the projectors $f_{vp}(\cdot )$ and $f_{tp}(\cdot )$ .

Next, we apply L2 normalization to both features to ensure consistency, obtaining:

$v_i^{``}=\frac{v_i^{`}}{\left|\right|v_i^{`}{\left|\right|}_2}$and $t_i^{``}=\frac{t_i^{`}}{\left|\right|t_i^{`}|{|}_2}$. Then$v_i^{``}$ and text feature$t_i^{``}$ along the feature dimension to form a vision-language joint representation, denoted as:$g_i=Concat\left(v_i^{``},t_i^{``}\right)$. This joint feature is then fed into a classifier, which consists of multiple fully connected layers, dropout layers, and ReLU activation functions. The classification process is formulated as:

${\displaystyle {\displaystyle {\hat{y}}_i=f_{\text{fc}}\left(g_i,{\theta }_{\text{fc}}\right)}}$(5)

where ${\theta }_{fc}$ represents the learnable parameters of the classifier.

Model Training

The internal dataset is divided at the patient level into training, validation, and test sets in a ratio of 8:1:1, strictly following the principle of data isolation. Regarding the loss function during training, we use the cross-entropy loss function ${\mathcal{L}}_{\mathcal{c}\mathcal{e}}$ and the contrastive loss function${\mathcal{L}}_{\mathcal{c}\mathcal{t}},$ which are formulated as follows:

${\displaystyle {\displaystyle {\displaystyle {\mathcal{L}}_{CE}=-\sum _{i=1}^c{y}_i\log ({\hat{y}}_i)}}}$(6)

${\displaystyle {\displaystyle {\displaystyle {\mathcal{L}}_{CT}=-\log \frac{\exp \left(\frac{sim(z_i,z_j)}{\tau }\right)}{\sum _{k=1}^{2N}{\mathbf{1}}_{[k\ne i]}\exp \left(\frac{sim(z_i,z_k)}{\tau }\right)}}}}$(7)

In ${\mathcal{L}}_{\mathcal{c}\mathcal{e}}$, C represents the total number of classes, $y_i$ denotes the ground-truth class label in one-hot encoding, and $\widehat{y_i}$ is the predicted probability for class I. This loss function aims to minimize the difference between the model’s predicted distribution and the true distribution, thereby guiding parameter updates and optimization. In ${\mathcal{L}}_{\mathcal{c}\mathcal{t}}$, $sim(·)$ represents cosine similarity,$\mathrm{}{z}_i$ and $z_j$ are positive sample pairs, $\tau$ is the temperature parameter, and N is the number of negative samples. The final loss function ${\mathcal{L}}_{loss}$ is formulated as follows:

${\displaystyle {\displaystyle {\mathcal{L}}_{loss}={\mathcal{L}}_{ce}+c\times {\mathcal{L}}_{ct}}}$(8)

where c=0.1. All training processes and experiments are conducted on a dedicated server equipped with 4 NVIDIA A6000 GPUs with a total of 196GB of VRAM. The system runs on Ubuntu 20.04.5 LTS, and the model is implemented using PyTorch 3.9.0 and Scikit-learn 1.3.1. In this study, we use the AdamW optimizer to optimize VLMF-Net, with an initial learning rate set to 0.00001. A warm-up strategy and cosine learning rate scheduling are adopted to dynamically adjust the learning rate. VLMF-Net is trained for a total of 80 epochs.

Ethical Considerations

This study was approved by the Ethics Committee of the First Affiliated Hospital of Sun Yat-sen University (approval number [2023]755‐1). Informed consent was waived by the institutional review boards of all participating hospitals due to the study’s retrospective design. We implemented stringent measures to protect the privacy of all participants by anonymizing all collected data to remove any personally identifiable information. Throughout the manuscript preparation, we diligently avoided disclosing any details that could reveal the identity of participants. Furthermore, no compensation or indemnity was required as the study did not cause any losses to participants beyond the necessary clinical diagnostic and therapeutic measures.

Results

Result of Our Model

To demonstrate the effectiveness and advantages of our proposed VLMF-Net, we selected 4 classic models as baseline models and performed comparisons on 2 datasets. The 4 models include vision-and-language transformer (VILT) [29], contrastive language-image pretraining (CLIP) [30], bootstrapping language-image pretraining with frozen image encoders and large language models (BLIP-2) [25], and a large-scale image and noisy-text embedding (ALIGN) [31], and the 2 datasets refer to the internal and external datasets mentioned earlier. Tables 2 and 3 present the average results of 5 trials for the 4 baseline models and our proposed model on the internal and external datasets, respectively. Figure 3 shows the receiver operating characteristic curves and corresponding area under the curve values of different models on internal and external datasets. As shown in Table 2, on the internal dataset, our method demonstrates significant advantages, achieving the following evaluation metrics: accuracy (0.776), precision (0.820), and F₁-score (0.776). Notably, compared to the second-best model, our method achieves significant improvements of 0.061, 0.032, and 0.046 in accuracy, precision, and F₁-score, respectively. In terms of class-wise recall, VLMF-Net achieves recall rates of 0.754 and 0.803 for VCD and GC, respectively. Compared to other models, VLMF-Net shows significant improvements across multiple evaluation metrics, indicating its superior ability in recognizing glottic cancer. In addition, as shown in Table 3, on the external dataset, our method also demonstrates significant advantages, achieving accuracy (0.739), precision (0.828), and F₁-score (0.737), with improvements of 0.046, 0.053, and 0.046 over the second-best model, respectively. In terms of class-wise recall, our model achieves recall rates of 0.701 and 0.793 for VCD and GC, respectively. Compared to other models, there is also a significant improvement, which further demonstrates that VLMF-Net possesses excellent generalization ability and robustness. Moreover, as shown in Figure 3, our model demonstrates significant advantages in area under the curve metrics on both internal and external datasets. On the internal dataset, it outperforms the second-best model by 0.026, and on the external dataset by 0.012.

‎

Ablation Studies

To validate the effectiveness of our multimodal approach for early diagnosis of GC, we designed an ablation study to systematically evaluate the performance differences between models using single-modal and multimodal inputs: (1) M1: a single-modal model using only laryngoscopy images; (2) M2: a single-modal model using only laryngoscopy diagnostic reports; (3) M3: a multimodal model combining images and text, but without Q-Former. (4) M4: a multimodal model combining images and text with Q-Former. The experimental results, as shown in Table 4, indicate that the multimodal model M4 significantly outperforms both single-modal models across all evaluation metrics. Specifically, compared to the best single-modal model, M4 achieves improvements of 0.098 in accuracy (0.776), 0.106 in precision (0.820), 0.100 in recall (0.779), and 0.098 in F₁-score (0.776). This demonstrates that the multimodal data fusion strategy effectively integrates visual features and textual semantic information, significantly enhancing the diagnostic performance of the model.

In addition, to verify the effectiveness of the Q-Former module, we designed ablation experiments related to the Q-Former module. The experimental results are shown in Table 4. M4 outperforms M3 in all indicators. Specifically, M4’s accuracy (0.776) increased by 0.054, precision (0.820) increased by 0.112, recall (0.779) increased by 0.056, and F₁-score (0.776) increased by 0.054. This indicates that the Q-Former module effectively reduces the impact of cross-modal differences.

Visualization for Model Prediction

To enhance the interpretability of the VLMF-Net model’s predictions, we used the Grad-CAM (Gradient-Weighted Class Activation Mapping) [32] algorithm to generate class activation heatmaps. The visualization results are shown in Figure 4. In the heatmap, we can see that VLMF-Net focuses on the lesion area of the patient when analyzing the laryngoscopy image. This indicates that the model is able to correctly classify based on the features of the lesion area, enabling early diagnosis of GC.

‎

Discussion

Main Findings

In our study, we developed a novel model for the early diagnosis of GC named VLMF-Net, which leverages multimodal fusion technology guided by vision-language information. When tested on an internal dataset, VLMF-Net achieved accuracy, precision, recall, and F₁-score of 0.776, 0.820, 0.779, and 0.776, respectively, outperforming all baseline methods. These results demonstrate the feasibility and effectiveness of VLMF-Net in the early diagnosis of GC. Ablation studies further reveal that VLMF-Net significantly outperforms unimodal models. By integrating both laryngoscopic images and clinical text reports, VLMF-Net captures complementary diagnostic information, thus mitigating the risk of information loss inherent in single-modality systems. Notably, the textual modality meaningfully guides the extraction of image features; for instance, textual cues explicitly describing the lesion’s location (eg, “anterior-middle segment of the left vocal cord”) can help the model focus on the relevant visual regions. This form of cross-modal interaction may enhance the model’s ability to detect subtle pathological patterns that might otherwise be overlooked.

Moreover, Grad-CAM–based visualizations demonstrate that VLMF-Net consistently attends to clinically significant lesion areas. These attention heatmaps show strong alignment between the model’s focus and expert-defined pathological regions, enhancing interpretability. Such interpretability is crucial in clinical contexts, where transparency in decision-making processes fosters trust and facilitates integration of AI systems into diagnostic workflows.

Finally, these findings validate the design philosophy of VLMF-Net and highlight the broader potential of multimodal fusion strategies in medical AI. In particular, the ability to synthesize visual and contextual clinical information allows for more robust and informed diagnostic decisions. This work not only advances the state of the art in GC diagnosis but also lays a foundation for extending vision-language multimodal techniques to other complex diagnostic tasks where rich multimodal data is available.

Deep Learning Challenges in Early GC Diagnosis

Early GC lesions are usually small and exhibit complex morphological characteristics, making it challenging for deep learning models to capture fine-grained lesion features. As a result, important details related to the lesions may be missed, significantly affecting the model’s diagnostic accuracy. Furthermore, most existing mainstream models rely solely on laryngoscopic images as input. While these models have achieved some progress in a unimodal setting, they overlook information from other modalities, which is often beyond the reach of image-based models [15-19]. This limitation constrains the model’s comprehensive understanding of lesion characteristics and ultimately affects diagnostic accuracy.

Strengths and Limitations

To the best of our knowledge, this is the first study to apply multimodal techniques to the early diagnosis of GC. In our model, we fully utilize patients’ laryngoscopy reports and laryngoscopic images to extract relevant information about GC, enabling the model to comprehensively understand the patient’s condition and thereby improve its performance. However, this study still has some limitations. First, the VLMF-Net, based on a pretrained large language model, relies heavily on powerful computational resources, particularly high-performance GPUs, during training, which may result in slower training and inference speeds. Second, the data used in this study underwent quality checks, but in real-world scenarios, more complex situations may arise, such as poor image clarity due to imaging devices or challenging shooting angles, which could affect the model’s diagnostic accuracy.

Conclusions

In this paper, we propose a VLMF-Net for the early diagnosis of GC. Extensive experiments on 2 datasets demonstrate that VLMF-Net achieves superior accuracy and robustness, effectively addressing the challenges of early GC diagnosis.