A total of 226 participants, including 166 (73.5%) women and 60 (26.5%) men aged between 24 and 87 years, were included in this prospective, clinical study. Endoscopic HSI data were acquired at the Department of Oral and Maxillofacial Surgery, Facial Plastic Surgery, University Medical Center. The examined participants represented a heterogeneous group of patients from the oral and maxillofacial surgery outpatient clinic. Patients with macroscopic abnormalities of the oral mucosa, premalignant lesions, or tumors were excluded from the study. All participants provided informed consent prior to the procedure and data collection. The individual pictured in Figure 1 has provided permission for their image to be used in this publication. This study was approved by the local ethics committee of Rhineland-Palatinate (registration number 2021‐16158) and was conducted in accordance with the code of ethics of the World Medical Association (Declaration of Helsinki).

The HSI datasets were acquired using a state-of-the-art endoscopic HSI sensor system (TIVITA Mini^® camera system, Diaspective Vision GmbH). This system detects 100 wavelengths in the range of 500‐1000 nm, offering a bandwidth of 5 nm [34]. The device uses the pushbroom principle, enabling chemical component detection based on light absorption and reflection behavior. During operation, light enters the optical system of the spectrometer through the lens, where it is collimated and separated into individual wavelengths using a transmission grating. The separated light then passes through a second optical system before reaching the sensor of the connected complementary metal-oxide-semiconductor camera. The spectrometer directly detects the spatial direction and width of the object being scanned (Y-axis) while the second spatial direction and length of the object (X-axis) are determined through the continuous mechanical movement of the light entry gate within the scanning unit. This process generates a 3D data cube that includes a spectral dimension (λ), capturing complete tissue spectra for each pixel within the wavelength range of 500‐1000 nm [35].

Standardized measurement protocols were followed during image acquisition, maintaining a consistent distance of 7‐10 cm to ensure high-quality and distinct image data (Figure 1). Measurements were performed in an examination room under dimmed lighting conditions to ensure uniformity and comparability. A total of 226 participants aged between 24 and 87 years, including 166 (73.5%) women and 60 men (26.5%), were prospectively recruited at the outpatient department of the clinic of Oral and Maxillofacial Surgery, University Medical Center, where all individuals underwent routine clinical evaluation and inspection. Inclusion was limited to individuals without clinically apparent mucosal lesions, premalignant conditions, or malignancies, ensuring a representative sample of nonpathological oral tissue. Informed consent was obtained from all participants prior to data acquisition. The recruitment strategy was designed to capture a broad range of healthy oral tissue presentations to ensure the generalizability of HSI data across different anatomical sites and patient demographics. For each patient, 5 images were taken, including views of the right and left cheeks, the palate, the back of the tongue, and the closed row of teeth. The data were subsequently pseudonymously archived using camera-specific software tools.

The Comprehensive Oral Health Hyperspectral Dataset (comprehensive, annotated endoscopic HSI dataset of the oral cavity) primarily comprises RGB (red, green, and blue) and endoscopic HSI data with corresponding annotations. These components in the presented dataset are each important to address the fundamental challenges in DL for semantic segmentation [36], providing robust and effective model training. The RGB component offers standard visual spectrum imagery of the oral cavity, serving as a foundational baseline for comparison with more sophisticated imaging modalities like raw and processed endoscopic HSI data.

Individual annotations of the oral cavity’s anatomical sites were manually performed using an image annotation tool (RectLabel Pro version 2024.06.07, Ryo Kawamura; Tokyo, Japan). To ensure high-quality image data annotation, several key features were implemented. First, detailed annotation guidelines were developed to ensure reproducible results. Comprehensive training sessions and practical exercises for annotators were conducted, followed by feedback rounds to enhance their skills. Each image was annotated by at least 2 independent specialists to minimize subjective errors. Regular quality checks were performed through sample reviews by senior specialists to detect inconsistencies. Continuous feedback was given to annotators, and guidelines were regularly updated based on this feedback. Interannotator analysis of 2 segmentation annotation sets (“original” vs “new”), each assigning a class label (out of 20 possible objects plus background) to every pixel, was carried out additionally for internal quality assurance. Agreement was quantified using metrics that capture both overall and class-specific consistency: (1) Pixel Accuracy, (2) Cohen Kappa (including and excluding background), and (3) Dice Coefficients, which are mathematically equivalent to the F₁-score in the binary case. To ensure a realistic representation of everyday clinical practice, the annotated structures marked for subsequent analysis corresponded to the individual intraoral site and situation, including “clutter (n=752),” “blood (n=3),” “brackets (n=15),” “floor of mouth (n=43),” “gingiva (n=511),” “implant (n=10),” “lip (n=603),” “mucosa (n=745),” “palate (n=539),” “prosthesis (n=65),” “reflection (n=106),” “retractor (n=702),” “suture (n=2),” “telescopic crown (n=28),” “tongue (n=587)” and “tooth (n=681),” with a number of distinct classes varying from a minimal count of 2 for “suture” to a substantial count of 752 for “clutter”.

To minimize diagnostic errors, undersegmentation of anatomical structures was avoided, and experienced clinical experts reviewed each annotation to ensure that all clinically relevant features were thoroughly labeled. This conservative approach occasionally led to small unannotated gaps between each class to prevent the merging of different anatomical sites. The digital annotations in XML format provided a semantic comprehension of the image data, which are essential for the application and training of machine learning and computer vision to differentiate various oral site conditions based on the RGB and endoscopic HSI components (Figure 2).

Preprocessing further involved the conversion of the original hyperspectral data from .DAT format to the NumPy format (.npy) to facilitate interoperability, reproducibility, and ease of use for DL models. Regarding data augmentation, spatial transformations, such as image rotations to enhance model robustness, were used. The addition of noise and blur was intentionally avoided, as these operations could introduce spectral distortions that are not physically plausible in the HSI context.

Given the specific goal of oral structure differentiation, classes with low incidence, such as suture, blood, implant, brackets, telescopic crown, prosthesis, reflection, and floor of mouth were deemed less relevant and excluded from further analysis to prevent skewing the learning process and improve the model’s ability to generalize to more common and diagnostically relevant structures. By focusing on higher-incidence classes, such as palate, retractor, mucosa, tongue, tooth, lip, and gingiva, the training process is optimized, enhancing the model’s efficiency and effectiveness across diverse oral structures. To achieve representative results for the annotation, an additional interannotator analysis and validation was carried out to detect possible person-dependent deviations.

To ensure user-friendly analysis of the hyperspectral dataset and reliable collaboration in the research community, the dataset has been specifically optimized for the Python ecosystem (Python Software Foundation 2023, Python Language Reference, version 3.10) instead of the proprietary software required to load endoscopic HSI data. Regarding scientific computation, Python emerges as the predominant programming language due to its extensive libraries and robust community support. After acknowledging this prevalent preference and recognizing the NPY file format as the most straightforward option for loading data into NumPy, a pivotal library for numerical computations in Python [37], all elements of the comprehensive, annotated endoscopic HSI dataset of the oral cavity (“RGB,” “Annotations,” “HSI_Data_Files”) were archived in the “.npy” format. The NPY format offers several advantages: It is compact and efficient for loading and saving and supports a wide range of data types. This standardization eliminates the complexity and potential incompatibilities associated with proprietary formats. In order to provide high-quality, detailed annotations of the complex intraoral structures, digital annotations were elaborated in XML format. To enhance utility and facilitate integration into researchers’ workflows, a semantic map of the oral cavity was generated directly from the XML datasets for each image. These maps essentially consisted of binary or multiclass masks, delineating the categorization of each pixel based on the detailed information provided in XML. However, the creation of these maps is an automated process that translates the complex structured information of the XML annotations into a simple but comprehensive format that can easily be integrated into machine learning models to differentiate tissue types based on the values of each pixel corresponding to the specific annotated category. The workflow in short: First, eHSI endoscopic HSI data are loaded, and semantic information is extracted from the XML file. This information is then used to generate a semantic map of the oral cavity, where each pixel value corresponds to a specific annotated category, such as different tissue types or anatomical sites. The resulting semantic map provides comprehensive, pixel-wise annotations of the frame, ready for application in segmentation algorithms.

For an initial assessment of the spectral data, a coefficient of variation (CV) was used as a statistical measure. The CV is a standardized measure for dispersion of a probability distribution or frequency distribution and defines the ratio of the SD to the mean, expressed as percentage. It is particularly useful in the context of HSI, as it allows the comparison of variation between different classes and bands despite the different mean intensities. The formula used for the coefficient of variation is:

where σ is the SD of the dataset and μ is the mean of the dataset.

This study adapted several state-of-the-art models and their performance assessing semantic segmentation in the context of oral health diagnostics. These models included DeepLabv3 [38-40], as well as fully convolutional network (FCN [41]) and pyramid scene parsing network (PSPNet [42]), each tested with ResNet-50 and ResNet-101 [43], backbones. In addition, PSPNet using VGG16 [44] and U-Net [45] models using both EfficientNet-B0 [46] and ResNet-50 backbones were evaluated. Each model was optimized for processing our dataset, which features challenging anatomical structures, enabling a comprehensive analysis of their suitability for handling the variability inherent in endoscopic HSI data for oral health diagnosis. The models were trained over 50 epochs using 70% of the comprehensive, annotated endoscopic HSI dataset of the oral cavity test set, with the remaining 30% reserved for evaluation. This split allowed for a thorough assessment of the models’ ability to generalize to unseen data. Each model’s performance on endoscopic HSI data was evaluated using Precision, Recall, and the F₁-score, providing a comprehensive measure of their effectiveness in segmenting and classifying the various anatomical structures present in the dataset.

The study was approved by the local ethics committee of Rhineland-Palatinate (registration number: 2021-16158) and was conducted in accordance with the protocol and in compliance with the moral, ethical, and scientific principles governing clinical research as set out in the Declaration of Helsinki of 1975 as revised in 1983. Informed consent was obtained from all participants involved in the study. This study received approval from the Ethics Committee of the Medical Association of Rhineland-Palatinate (reference number 2021-15858). All procedures adhered to institutional and national ethical standards and were conducted in accordance with the Declaration of Helsinki. Participants were provided comprehensive information regarding the study’s nature, purpose, procedures, data usage, and the potential publication of anonymized images or data related to their participation. The privacy and confidentiality of all participants were rigorously protected. No identifying information, including names or hospital IDs, has been included in the manuscript. Images containing identifiable features were excluded except for Figure 1, which depicts the corresponding author (PR), who has provided written consent for the use of their image in this publication. No financial or material compensation was offered or provided to the participants in this study.

A descriptive statistical analysis of the hyperspectral dataset was performed to describe the distribution and variability of the data. Mean values and SD across the spectral bands were analyzed for each class delineated in the hyperspectral images. The objective of the statistical data evaluation was to accurately represent the central tendencies and variabilities of spectral signatures, which are indicative of various anatomical and pathological entities in oral health. Mean values and SDs for all classes across all spectral bands were presented in a consolidated diagram (Figure 3).

The results indicated that the CV for the majority of classes, across most spectral bands, substantially exceeded the commonly recognized threshold of 15%, which is generally considered high. In contrast, a CV below 5% would have denoted homogeneous data, while values between 5% and 15% would have indicated moderate uniformity. The consistently elevated CV values observed across all classes thus underscored significant spectral diversity within each class (Figure 4).

The DeepLabv3 model with the ResNet-50 backbone achieved solid overall performance, with an F₁-score of 0.855, performing especially well on the segmentation of mucosa, retractor, and tooth. However, it exhibited moderate performance on gingiva and lip, with F₁-scores of 0.753 and 0.709, respectively. Switching to the deeper ResNet-101 backbone improved the overall performance slightly, with an F₁-score of 0.857, particularly enhancing the model’s ability to segment clutter and gingiva (Table 1).

Both FCN-ResNet-50 and FCN-ResNet-101 demonstrated robust segmentation performance, achieving overall F₁-scores of 0.862 and 0.861, respectively (Table 2). They excelled in segmenting retractor (F₁-score=0.942 for both), tooth (F₁-score=0.910), palate (F₁-score=0.890), and mucosa (F₁-score≥0.912). However, lip and gingiva posed challenges for both backbones, reflected by lower F₁-scores around 0.72 and 0.77, respectively. Although switching to the deeper ResNet-101 backbone yielded marginal improvements in certain classes, the overall performance remained comparable between the 2 architectures.

PSPNet models showed slightly lower performance, with the ResNet-50 backbone achieving an F₁-score of 0.837 and the VGG16 backbone scoring 0.808. While these models handled classes, such as retractor and tooth, relatively well, they faced difficulties in the segmentation of gingiva and lip, particularly when using VGG16. The lower overall performance of PSPNet VGG16 compared to ResNet-50 reflects the influence of the backbone on the segmentation outcomes (Table 3).

U-Net (EfficientNet-B0) and U-Net (ResNet-50) both demonstrated robust segmentation performance, attaining overall F₁-scores of 0.867 and 0.840, respectively (Table 4). They excelled particularly at segmenting retractor (F₁-score=0.941 vs 0.927), palate (F₁-score=0.909 vs 0.881), and mucosa (F₁-score=0.920 vs 0.901). Tongue and tooth segmentation also performed well, with F1-scores exceeding 0.85 for both models. However, both networks faced challenges in segmenting lip (F₁-score=0.755 vs 0.695) and gingiva (F₁-score=0.751 vs 0.721), indicating opportunities for further refinement in these classes.

Figure 5 demonstrates that DeepLabv3 (ResNet-101) and U-Net (EfficientNet-B0) outperform other models across most tissue classes, with lower F₁-scores consistently observed for lip and gingiva segmentation.

The analysis of segmentation annotations revealed an overall pixel accuracy of 80.84%, indicating a high level of interannotator agreement. Cohen Kappa, a more robust measure of interannotator agreement, was 77.56% when including the background class, indicating substantial agreement. When the background was excluded, Kappa increased to 82.91%, highlighting even stronger agreement for the segmentation of foreground structures. Dice coefficients per class ranged from near-zero (in classes with minimal or inconsistent labeling) to 93% in well-defined classes, where the overlap between annotations was nearly perfect. Since the Dice Coefficient measures spatial overlap between segmentation masks, high Dice scores (above 90%) indicate excellent agreement, whereas lower values suggest potential ambiguity or underrepresentation of certain structures.

The objective of this study was to establish a digital map of the oral cavity using endoscopic HSI in conjunction with advanced machine learning techniques and therefore to develop a comprehensive endoscopic HSI dataset. This approach aimed to enable reliable and automated differentiation of various tissue types and objects based on the spectral data acquired from the endoscopic HSI system. Despite the continuous interest in faster, minimally invasive diagnostic techniques, methods, such as brush biopsies and in vivo fluorescence procedures have faced challenges in establishing themselves as reliable alternatives due to their comparatively lower sensitivity and specificity [2,4,5,9]. Modern methods, such as endoscopic HSI, represent a state-of-the-art, innovative approach in the field of automated image and tissue classification. In our preliminary ex vivo studies, we have already demonstrated that HSI can differentiate between various tissue types and states based on specific wavelength patterns [20,47]. However, to effectively differentiate between pathological and healthy tissue conditions in vivo, a substantial dataset of hyperspectral signatures from healthy tissues is essential [47].

The study introduces a comprehensive collection of 1,130,751 endoscopic HSI–cubes of healthy oral mucosa, captured in vivo from various angles, creating a representative digital map that includes relevant tissues and objects of the oral cavity. This dataset forms a representative digital endoscopic HSI map, encompassing relevant tissues and structures of the oral cavity. By establishing a robust reference for healthy tissue, this dataset lays the groundwork for advancing HSI-based diagnostics, particularly in the identification of premalignant and malignant mucosal lesions and the precise definition of tumor resection margins.

The findings revealed elevated CV across most classes, indicating substantial spectral diversity in endoscopic HSI data. This high variability poses challenges for threshold-based classification methods, as intraclass variation can lead to class overlap. However, this diversity also holds valuable information that, when leveraged by advanced computational models, can enhance tissue classification and pathological anomaly detection with high accuracy.

To address this complexity, this study evaluated several state-of-the-art segmentation models, including DeepLabv3, FCN, PSPNet, and U-Net, with different backbones, such as ResNet-50, ResNet-101, VGG16, and EfficientNet-B0. These architectures were chosen for their effectiveness in semantic segmentation, balancing feature extraction capability, computational efficiency, and global context recognition. DeepLabv3 was selected for its ability to handle scale variability and capture fine details while FCN served as a strong baseline due to its foundational role in image segmentation. PSPNet was included for its strong capability in capturing global context, which is crucial for recognizing complex structures in medical imaging. U-Net, widely used in medical imaging, was chosen for its ability to achieve accurate segmentation even with limited data.

The choice of backbones was guided by their specific strengths. ResNet-50 and ResNet-101 were selected for their robust feature extraction, VGG16 for its simplicity and high-resolution detail, and EfficientNet-B0 for its optimized architecture that balances performance and computational efficiency. By incorporating models with varying depths and parameter complexities, this study ensures a comprehensive evaluation of segmentation performance while maintaining a focus on efficient training and inference times. The integration of the DeepLabv3 model with the ResNet-50 as well as with the ResNet-101 backbone offers a robust semantic segmentation approach for endoscopic HSI data interpretation. The primary modification to all the models, including DeepLabv3, FCN, PSPNet, and U-Net (with backbones, such as ResNet-50, ResNet-101, EfficientNet-B0, and VGG16), involved adapting the first convolutional layer to handle the variable number of channels in the dataset. This adjustment was essential to accommodate the multidimensional spectral data of the HSI dataset, as opposed to the standard 3-channel RGB imagery. This modification enables the models to use the unique spectral information in the HSI data cubes beyond the visible spectrum. Despite these adjustments, the deep residual learning architectures of ResNet-50 and ResNet-101 retain their capacity to extract high-level features—an essential aspect of HSI data analysis [43]. These backbones excel at identifying subtle spectral patterns crucial for accurate disease detection and classification. Leveraging atrous convolution and atrous spatial pyramid pooling, the DeepLabv3 model efficiently captures multiscale information [48]. This capability facilitates precise image segmentation, an essential feature for diagnosing oral health conditions with subtle and overlapping visual signatures. Likewise, FCN and PSPNet, with their robust architectures, and U-Net, recognized for its effectiveness in medical imaging, also benefited from the capacity to process and interpret multidimensional spectral data.

The endoscopic HSI procedure implemented in this study achieves results that are comparable to those obtained in preliminary HSI-ex vivo trials conducted by our research group in 2021 [47]. In this study, using a similar approach with a lightweight 6-layer deep neural network containing 10,445 parameters trained over 4000 epochs, tissue samples of fat, muscle, and oral mucosa could be differentiated with an overall class accuracy of over 80%. Similarly, in a study by Ma et al, tissue from various organs, including the kidney, liver, lung, muscle, salivary gland, and spleen, was identified using automated polarized HSI with an accuracy of up to 87% [49]. Furthermore, a deep convolutional neural network established by Poonkuzhali et al [50] could accurately identify brain tissue with an F₁-score precision of 97.3% using HSI in a recent study conducted in 2023. Unlike previous studies that focused on ex vivo tissue samples and organ-specific HSI data [47], the present work introduces the first large-scale in vivo annotated endoscopic HSI dataset of the oral cavity acquired under realistic clinical conditions. While earlier studies demonstrated the technical feasibility of HSI-based tissue differentiation, these approaches often lacked anatomical complexity, semantic annotations, and clinical variability. This study addresses these deficiencies by providing a large annotated in vivo HSI dataset and evaluating multiple DL models especially adapted for HSI data analysis. This provides the basis for clinically applicable segmentation of oral tissues and paves the way for future studies involving pathological lesions.

These findings affirm that HSI, when integrated with appropriate modalities, can serve as a reliable tool for differentiating various tissues in both in vivo and ex vivo setups. In oncological surgery, fast, reliable, and minimally invasive diagnosis of pathological tissue conditions is of paramount importance. Numerous studies in this field have shown that HSI can dependably differentiate between tumor and healthy tissue by analyzing histopathological sections and their spatial-spectral features. Moreover, the combination of HSI and DL has shown superior results compared to the use of RGB images and conventional support-vector-machine approaches [51-56].

In this study, the established DL and neural network approach was able to accurately identify different types of oral mucosa with an overall precision of approximately 91%. DeepLabv3 (ResNet-101) and U-Net (EfficientNet-B0) emerged as the top performers, demonstrating robust segmentation across key anatomical classes. While all models could benefit from further refinement in gingiva and lip segmentation, their consistent accuracy in identifying retractors and teeth underscores the strength of the dataset. While there is no universally accepted threshold for clinical applicability, F₁-scores above 0.85 are generally considered promising in similar biomedical imaging tasks. Scores in this range suggest that the method may already approach a level of accuracy relevant for clinical decision-making, though further validation in real-world settings would be required. Notably, U-Net (EfficientNet-B0) exhibited exceptional segmentation of retractors, mucosa, and teeth, indicating its strong potential for medical image segmentation applications. The results demonstrate the dataset’s suitability for training advanced neural networks, particularly in challenging medical segmentation tasks.

Limitations of our study specifically include the acquisition of a higher amount of data, which depended heavily on the individual patient volume of the clinic conducting the study. This represents a major challenge, particularly when aiming to document various oral mucosal lesions and precancerous mucosal conditions. To address the issue of data scarcity, the raw data generated in this study will be made publicly available. This initiative aims to provide other research groups with the opportunity to use and build upon the dataset for further investigations. Furthermore, while the study cohort was intentionally designed to include a broad cross-section of patients from a university-based outpatient clinic, selection bias cannot be fully excluded. As recruitment was confined to a single university clinic, the cohort may not fully reflect the demographic and clinical variability found in primary care or the general population. Other limitations include variable lighting conditions, motion artifacts, and spectral overlaps, as well as overexposure and underexposure. Overexposure can occur due to various factors, such as the variability in patient anatomy, the movement during image capture, or the fluctuation of lighting conditions in an operating room. While overexposed images are typically regarded as artifacts and excluded from datasets, our approach incorporates them, acknowledging that such occurrences are an inherent part of clinical practice. Understanding that these instances present both challenges and opportunities for advanced image processing techniques, we have carefully indexed these overexposed images for further analysis. This approach enhances the resilience and adaptability of the models trained on this dataset, ensuring they perform effectively in real clinical environments. By incorporating overexposed images, the models are better equipped to handle the full spectrum of data variability, including common environmental factors. This combination highlights the substantial potential of endoscopic HSI data to significantly enhance the precision and reliability in detecting and categorizing healthy and pathological oral conditions. This convergence not only paves the way for significant advancements in diagnostic methodologies but also holds the potential to substantially enhance patient care. External validation is currently planned as part of future work, particularly for mucosa segmentation. We intend to validate our model on an ex vivo dataset that includes mucosa samples captured under different acquisition conditions. At this stage, the focus of this study was the initial validation on the internally collected dataset. Corresponding follow-up studies are currently in preparation and will include HSI of pathological tissue conditions, such as oral mucosal lesions and neoplastic changes, to assess the transferability and diagnostic robustness of the proposed models in clinically relevant scenarios. By enabling earlier detection and more precise characterization of oral health conditions, this approach facilitates the development of more effective treatment strategies, ultimately leading to improved patient outcomes and a higher standard of care.

This study presents the first large-scale in vivo annotated dataset of the oral cavity using endoscopic HSI under realistic clinical conditions. By combining hyperspectral datasets and DL-based segmentation, it was possible to demonstrate the feasibility of automated, noninvasive tissue classification across important anatomical intraoral structures. Comparative analysis reveals DeepLabv3 and U-Net as robust architectures for oral tissue classification. The comprehensive dataset consequently provides a sufficient foundation for future work on pathological tissue detection, intraoperative margin assessment in oncology, and early, individualized diagnostics in oral medicine.