Diagnosing Rare Genetic Diseases From Facial Images Using Visual Prompting With Multimodal Large Language Models: Diagnostic Accuracy Study

Introduction

The combined number of people affected by rare genetic diseases is substantial [1]. Clinical manifestations are highly heterogeneous, and patients often undergo prolonged, repeated examinations before receiving a definitive diagnosis [2,3]. Many diseases are associated with recognizable facial phenotypes [4,5]. For example, in families with Crouzon syndrome [6], parents and children show highly overlapping facial features, sharing phenotypes such as mandibular prognathism, shallow orbits, proptosis, and exotropia. Specific facial phenotypes play a key role in the clinical diagnosis of diseases.

DeepGestalt [7] and GestaltMatcher [8] have shown that facial phenotypes can support the recognition of rare diseases through task-specific training or phenotype-similarity matching [9]. Large language models (LLMs) have also been explored for rare disease care [10,11]. DeepRare, for example, uses an LLM-powered multiagent framework to integrate clinical information, specialized tools, and medical knowledge, generating diagnostic hypotheses with traceable reasoning [12]. However, whether general-purpose multimodal LLMs (MLLMs) [13,14] can effectively prioritize rare genetic disease diagnoses from facial images alone remains an open question. One potential approach to supporting this task is to provide visual prompts that guide the models toward facial regions potentially relevant to disease discrimination [15-18].

Generative image models can produce realistic facial images, suggesting that they capture aspects of facial structure and appearance. This capability may support the localization of facial regions containing potentially relevant phenotypic features. These models can mark such regions in facial images and generate localized visual prompts for MLLMs, whose diagnostic utility can then be evaluated empirically.

To evaluate the facial image–based diagnostic performance of MLLMs and the effects of these prompting strategies, we used the GestaltMatcher Database (GMDB) [19] to compare 3 MLLMs across raw image, super-resolution, red-circle prompting, and 2-stage phenotype-mediated conditions, with mismatched-image controls and a model-independent random baseline. We assessed the clinical relevance and diagnostic utility of visual prompts through physician quality ratings and comparisons with expert annotations. Repeated-inference and uncertainty analyses were used to evaluate the stability of diagnostic accuracy and the relationships between model uncertainty measures and diagnostic hits. Analyses of prediction overlap and disease composition further characterized model outputs. The study workflow is presented in Figure 1. This study aimed to evaluate the diagnostic accuracy of MLLMs for rare genetic diseases using facial images, assess the benefits of image enhancement and visual prompting strategies, and examine the sensitivity of model predictions to patient-specific facial information.

‎

Methods

Study Dataset

This study used facial images and corresponding disease labels of patients with rare genetic diseases from the GMDB. On the basis of disease representation in the database, the dataset was divided into 2 subsets: GMDB-Frequent, comprising diseases represented by at least 6 patient images, and GMDB-Rare, comprising diseases represented by fewer than 6 patient images. A 10-fold sampling scheme was applied to GMDB-Rare. GMDB-Frequent included 882 image queries involving 275 diseases, whereas GMDB-Rare included 3864 image queries involving 306 diseases.

Image Processing and Experimental Conditions

CodeFormer (S-Lab, Nanyang Technological University) [20] was used to generate super-resolution images from the raw facial images. We used codeformer-pip v0.0.4 [21] with the official v0.1.0 pretrained weights [22]. The fidelity weight and upscale parameter were set to 0.8 and 2, respectively, with background enhancement and additional face upsampling disabled. We retained the implementation’s preprocessing and postprocessing routines: RetinaFace [23] (ResNet-50) detection, OpenCV-based 5-landmark alignment to 512×512 face crops, normalization to [−1, 1], and CodeFormer restoration, followed by inverse mapping and blending into the bilinearly resized image.

Qwen-Image-Edit-2511 [24] was then used to mark potentially abnormal facial regions in these images. Generation used 40 inference steps, true_cfg_scale=4.0, and guidance_scale=1.0. The red circles from the generated images were manually overlaid onto the corresponding CodeFormer super-resolution images to create the final visual prompts. This procedure preserved the facial content of the super-resolution images while incorporating the model-generated regional annotations.

Five input conditions were established for each patient facial image: (1) raw image diagnosis, in which the raw facial image was directly provided to the model; (2) super-resolution diagnosis, in which the CodeFormer-enhanced image was provided to the model; (3) red-circle prompting, in which the super-resolution image with manually overlaid red circles generated by Qwen-Image-Edit-2511 was provided to the model; (4) mismatched red-circle control, in which the current query image was replaced with a red-circle image from another patient in the same subset, with replacement restricted to the same fold in GMDB-Rare and the current query’s true diagnosis retained as the reference label; and (5) 2-stage image-to-phenotype diagnosis, in which the model first extracted a facial phenotype description from the current query’s red-circle image and then generated candidate diagnoses in a new conversation using only that text as input. All 5 conditions produced 50 candidate diagnoses ranked by likelihood. A random baseline was established by randomly selecting 50 diseases without replacement from the combined pool of 581 distinct diseases across GMDB-Frequent and GMDB-Rare, using Python’s random module with a fixed seed of 20260907. The same candidate list was used for all queries in both subsets.

Model Configuration and Diagnostic Inference

This study evaluated 3 MLLMs: Gemma-4-31B [25], Qwen3.6-35B-A3B [26], and Qwen3.8-27B [27]. Within each input condition, all 3 models received the same diagnostic instructions and generated 50 candidate disease names ranked by likelihood, returning 3 fields in JSON format: ranked_diagnoses, confidence, and insufficient_evidence. The confidence field represented the model’s self-reported probability (0%-100%) that the correct diagnosis was included in the complete list of 50 candidates. The insufficient_evidence field indicated whether the model considered the available information insufficient to support a reliable diagnosis. The red-circle prompting, mismatched red-circle, and expert-annotation conditions used the same prompt, with annotation sources masked to the models. The prompts used for facial abnormality annotation with Qwen-Image-Edit-2511 and for the 5 diagnostic conditions are provided in Table S1 in Multimedia Appendix 1.

To align with real-world clinical deployment and meet patient privacy and hospital data governance requirements, all 3 MLLMs and Qwen-Image-Edit-2511 were deployed locally as complete models on hardware comprising an NVIDIA RTX PRO 6000 GPU (NVIDIA Corp) and an Intel Xeon Platinum 8470Q CPU (Intel Corp). All 4 models were run in bfloat16 (BF16) precision without quantization. Details of the deployment environment are provided in Table S2 in Multimedia Appendix 1. The temperature was set to 0.3 for all inference requests, and thinking mode was disabled. Execution errors and invalid outputs, including an incorrect number of diagnoses, triggered retries, with a maximum of 10 retries per request. Nondiagnostic responses, such as abstentions, were retained as experimental outcomes and scored as incorrect for all top-k accuracy metrics. Duplicate diagnoses were retained in their original positions within the 50-item list. After candidate diagnosis generation, OntologyAligner [28] was used to map disease names to Mondo Disease Ontology (MONDO) IDs [29], harmonizing disease synonyms and naming variations and enabling comparisons between predictions and reference labels using standardized disease identifiers. OntologyAligner accomplishes the mapping through a 3-stage process: ontology-aligned retrieval, LLM candidate reranking, and selective hierarchy-guided refinement. OntologyAligner was configured with text-embedding-3-large for embedding, cosine similarity for retrieval, 20 candidate concepts per disease name, and gpt-5.6-sol for reranking at a temperature of 0. Input and output token use is summarized in Table S3 in Multimedia Appendix 1.

To assess mapping accuracy, 500 (2.1%) unique disease names were randomly sampled without replacement from the 23,482 unique names generated across the 3 MLLMs and 5 input conditions. Their MONDO assignments or no-match decisions were manually reviewed, and 97.2% (486/500) were judged correct. To assess sensitivity to simulated MONDO mapping errors, we used the raw image results from all 3 MLLMs in GMDB-Frequent and GMDB-Rare. At each perturbation level from 10% to 90%, in increments of 10 percentage points, we randomly selected the specified proportion of unique disease names. Each selected ID was replaced with a different valid MONDO ID chosen uniformly at random, and the replacement was applied to every occurrence of the same name. We repeated this procedure 500 times per level and recalculated the top-50 accuracy for each model and subset. The 0% level used the original mappings; for each nonzero level, we summarized the mean accuracy and the 2.5th and 97.5th percentiles across simulations.

Visual-Prompt Quality Evaluation

To assess the clinical relevance and localization quality of red-circle prompts, 2 physicians independently reviewed each red-circle image, with access to the corresponding disease label and facial phenotype information. Annotation quality was rated on a 3-level ordinal scale: poor, ordinary, or good. “Good” indicated that the annotated regions accurately highlighted facial abnormalities highly relevant to the target disease and had clear diagnostic value. “Ordinary” indicated that the annotated regions had some relevance but limited specificity or localization accuracy. “Poor” indicated weak relevance to the facial phenotypes associated with the target disease.

To assess the effect of annotation source on diagnostic accuracy, 100 queries were randomly selected from GMDB-Frequent, and 1 physician manually annotated potentially abnormal facial regions on the corresponding super-resolution images. Each of the 3 models performed diagnostic inference using both the expert-annotated images and the images with overlaid red circles generated by Qwen-Image-Edit-2511. Both conditions used identical diagnostic prompts and inference settings, and diagnostic accuracy was compared using a paired analysis of the same queries.

Repeated-Inference Analysis

To assess the stability of diagnostic accuracy across independent inference runs, 100 queries were randomly selected from each of GMDB-Frequent and GMDB-Rare, yielding 200 queries in total. Each of the 3 MLLMs completed 3 independent inference runs on the same queries under the raw image, super-resolution, and red-circle prompting conditions, using the same inference settings as the main experiment. Outputs from each run were saved separately, and disease names were mapped to MONDO IDs using OntologyAligner. Top-50 accuracy and its 95% CI were calculated for each model, input condition, and inference run.

Model Uncertainty Analysis

To assess the relationship between model uncertainty and diagnostic accuracy, we recorded self-reported confidence, evidence-sufficiency judgments, and diagnostic-list perplexity. Confidence represented the estimated probability (0%-100%) that the true diagnosis was included among the 50 candidates, while the insufficient-evidence flag indicated whether the model considered the available information insufficient for a reliable diagnosis. Perplexity was calculated by exponentiating the mean negative log probability of disease-name tokens and was used to quantify the model’s uncertainty about the generated sequence, with lower values indicating higher token probabilities:

Here, N is the total number of tokens across all disease names in the candidate diagnosis list; p_i is the probability assigned by the model to the ith disease-name token, conditional on the input and all previously generated content; and ln denotes the natural logarithm. We then analyzed the associations of self-reported confidence and perplexity with query-level top-50 hit status and compared diagnostic accuracy between queries classified by the models as having sufficient or insufficient evidence.

Prediction-Overlap Analysis

To assess the effects of input conditions on diagnostic predictions, we analyzed the overlap between sets of predicted diseases across conditions and the disease composition of top-1 predictions.

For each query, we calculated the Jaccard index between top-50 predicted disease sets under different input conditions, defined as the size of their intersection divided by the size of their union. Higher values indicate greater overlap between predicted disease sets. Disease-composition analysis compared the distribution of reference labels with that of the top-ranked predicted diseases for each model under each input condition. We separately identified the 5 most frequently occurring diseases and their proportions in the reference labels and each prediction group, grouping all remaining mapped diseases as “other.”

Statistical Analysis

Top-50 accuracy was the primary evaluation metric, with top-10, top-20, top-30, and top-40 accuracy also reported. The 95% CIs for accuracy, between-condition differences in accuracy, and other statistics were estimated using 10,000 bootstrap resamples with replacement, with the 2.5th and 97.5th percentiles of the bootstrap distribution serving as the lower and upper bounds. Accuracy was compared between input conditions using 2-sided exact McNemar tests, with P values adjusted for multiple comparisons using the Holm method. Uncertainty analyses were restricted to the raw image condition, and Spearman correlation analysis was used to assess the associations of model-reported confidence and diagnostic-list perplexity with top-50 hit status. Absolute between-condition differences in accuracy (percentage points), 95% CIs, and unadjusted and Holm-adjusted P values are provided in Multimedia Appendix 2, together with the corresponding correlation results.

Ethical Considerations

This study was not submitted to an ethics committee for review. The study used a legally obtained, publicly available dataset. According to article 32, item 1 of the Measures for Ethical Review of Life Science and Medical Research Involving Humans issued by the National Health Commission of the People’s Republic of China [30], research using legally obtained public data, or data generated through observation without interfering with public behavior, may be exempted from ethics review. Therefore, under this policy, not applying for ethics board review was appropriate, and no study protocol or application number is applicable.

Results

Diagnostic Performance Across Models and Input Conditions

Figure 2A presents the top-50 diagnostic accuracy for 882 queries in GMDB-Frequent, with accuracy ranging from 13.95% to 18.25% across the 3 models under 5 input conditions. Figure 2B presents the results for 3864 queries in GMDB-Rare, with corresponding accuracy ranging from 0.57% to 1.35%.

In comparisons of image processing and visual prompting conditions, the red-circle condition yielded numerically higher accuracy than the raw image condition in 5 of the 6 model-subset combinations and equal accuracy in 1. In GMDB-Frequent, accuracy for Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B changed from 15.87%, 14.51%, and 15.99% to 16.89%, 15.19%, and 16.44%, respectively, with P>.99 for all 3 comparisons. In GMDB-Rare, the corresponding accuracy changed from 0.91%, 0.91%, and 0.57% to 0.93%, 0.91%, and 1.27%, respectively. Qwen3.8-27B showed an increase of 0.70 percentage points (adjusted P=.009), while the other 2 models showed no significant changes (P>.99). The direction of accuracy changes for super-resolution vs raw images and for red-circle images vs super resolution varied by model and subset, with P>.05 for all comparisons.

In comparisons of patient-image matching, accuracy under the matched red-circle condition was numerically higher than under the mismatched red-circle condition across all 3 models and both subsets. The advantage ranged from 0.68 to 1.25 percentage points in GMDB-Frequent and from 0.26 to 0.49 percentage points in GMDB-Rare. All 6 comparisons yielded P>.05.

In comparisons of the 2-stage diagnostic baseline, image-to-phenotype yielded significant improvements over the raw image condition in 2 combinations: Gemma-4-31B in GMDB-Frequent, with accuracy increasing from 15.87% to 18.25% (an increase of 2.38 percentage points; adjusted P=.02), and Qwen3.8-27B in GMDB-Rare, with accuracy increasing from 0.57% to 1.35% (an increase of 0.78 percentage points; adjusted P=.003). Comparisons against the raw image condition in the remaining 4 combinations and against the red-circle condition in all 6 combinations yielded P>.05.

The random baseline achieved an accuracy of 12.02% in GMDB-Frequent and 8.15% in GMDB-Rare. In GMDB-Frequent, accuracy under both the raw image and red-circle conditions was numerically higher than the random baseline for all 3 models. For Gemma-4-31B, top-50 accuracy was significantly higher under the red-circle condition than under the random baseline (16.89% vs 12.02%, respectively; adjusted P=.02), while the other 5 comparisons yielded P>.05. In GMDB-Rare, the random baseline significantly outperformed all 3 models under both the raw image and red-circle conditions (all P<.001). Accuracy at different candidate-list cutoffs (k=10, 20, 30, 40, and 50) is presented in Figure S1 in Multimedia Appendix 1, and complete statistical results are provided in Multimedia Appendix 2.

Figure S2 in Multimedia Appendix 1 presents the sensitivity of raw-image top-50 accuracy to simulated MONDO mapping errors. Mean accuracy declined as the proportion of reassigned disease names increased across all 3 models and both GMDB subsets. At 10% reassignment, the mean decreases in accuracy across 500 simulations were 1.52 (SD 0.92), 1.34 (SD 1.04), and 1.50 (SD 1.30) percentage points in GMDB-Frequent and 0.08 (SD 0.14), 0.08 (SD 0.10), and 0.04 (SD 0.08) percentage points in GMDB-Rare for Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B, respectively.

‎

Visual-Prompt Quality and Diagnostic Benefit

Figure 3A presents the distribution of ratings independently assigned by 2 physicians to 4746 red-circle annotations generated by Qwen-Image-Edit-2511, categorized as poor, ordinary, or good. Ordinary was the predominant rating for both physicians, accounting for 48.9% (431/882) to 50.8% (448/882) of annotations in GMDB-Frequent and 61.6% (2381/3864) to 63.1% (2437/3864) in GMDB-Rare. The proportion rated good ranged from 31.5% (278/882) to 35.6% (314/882) in GMDB-Frequent and from 16.1% (624/3864) to 17.7% (683/3864) in GMDB-Rare. Overall agreement between the 2 physicians was 71.8% (3409/4746).

Figure 3B presents the quality-stratified analyses of GMDB-Frequent conducted separately using each physician’s ratings. In the good subgroup, Gemma-4-31B achieved significantly higher top-50 accuracy under the red-circle condition than under the super-resolution condition. In evaluation 1, accuracy increased from 17.27% to 23.02%, a gain of 5.76 percentage points (adjusted P=.005); in evaluation 2, accuracy increased from 17.20% to 21.66%, a gain of 4.46 percentage points (adjusted P=.048). All remaining stratified comparisons in GMDB-Frequent yielded adjusted P>.05.

Figure 3C presents the quality-stratified results for GMDB-Rare, where comparisons for all models across all quality subgroups yielded adjusted P>.05. Significant improvements in the quality-stratified analyses were confined to Gemma-4-31B in the GMDB-Frequent good subgroups defined separately by the 2 physicians’ ratings.

Under the red-circle condition in GMDB-Frequent, the good subgroup generally showed numerically higher accuracy. For Gemma-4-31B and Qwen3.8-27B in both evaluations and for Qwen3.6-35B-A3B in evaluation 2, accuracy in the good subgroup exceeded that in the corresponding ordinary and poor subgroups.

Figure 4 compares diagnostic accuracy between expert-annotated and model-generated circle images. A physician manually circled abnormal facial regions on 100 super-resolution images from GMDB-Frequent. Each of the 3 models received both expert-annotated and Qwen-Image-Edit-2511–generated red-circle overlays for the same set of queries. Under the expert-annotation condition, top-50 accuracy was 8.00% for Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B. Under the model-generated annotation condition, the corresponding accuracy was 8.00%, 6.00%, and 10.00%, respectively. Compared with model-generated annotations, expert annotations yielded equal accuracy for Gemma-4-31B, an increase of 2.00 percentage points for Qwen3.6-35B-A3B, and a decrease of 2.00 percentage points for Qwen3.8-27B. All 3 paired comparisons yielded adjusted P>.99. Detailed statistical results are provided in Multimedia Appendix 2.

‎

‎

Stability of Repeated Diagnostic Inference

To assess the stability of diagnostic performance across runs, we performed 3 independent inference runs on a fixed set of 200 queries, comprising 100 queries each from GMDB-Frequent and GMDB-Rare. Each model was evaluated under the raw image, super-resolution, and red-circle conditions using the same settings as in the main experiment.

Figure 5A presents the results for GMDB-Frequent. Across model-input combinations, the difference between the maximum and minimum top-50 accuracy over the 3 runs ranged from 1 to 4 percentage points. Across the 3 runs, mean accuracy for the raw image, super-resolution, and red-circle conditions was 13.67%, 14.33%, and 16.33%, respectively, for Gemma-4-31B; 13.33%, 13.00%, and 14.00% for Qwen3.6-35B-A3B; and 13.67%, 15.33%, and 14.67% for Qwen3.8-27B. Gemma-4-31B and Qwen3.6-35B-A3B achieved their highest mean accuracy under the red-circle condition, while Qwen3.8-27B achieved its highest mean accuracy under the super-resolution condition.

Figure 5B presents the results for GMDB-Rare. Across model-input combinations, the difference between the maximum and minimum accuracy over the 3 runs ranged from 0 to 3 percentage points. Gemma-4-31B achieved a mean accuracy of 1.00% under all 3 conditions. Across the raw image, super-resolution, and red-circle conditions, mean accuracy was 1.33%, 1.00%, and 2.00%, respectively, for Qwen3.6-35B-A3B and 1.33%, 1.00%, and 2.33%, respectively, for Qwen3.8-27B.

Across these 200 fixed queries, top-50 accuracy showed limited variation over the 3 independent inference runs. The low number of successful hits in GMDB-Rare means that changes in hit status for a small number of queries can affect accuracy.

‎

Model Uncertainty and Diagnostic Accuracy

To assess the relationship between uncertainty metrics and diagnostic accuracy, we asked the models to report diagnostic confidence and evidence sufficiency. We also recorded output-token log probabilities and calculated perplexity.

Figure 6A shows that the Spearman correlation coefficients between self-reported confidence and top-50 accuracy in GMDB-Frequent were 0.062, 0.057, and 0.037. Figure 6B shows the corresponding coefficients in GMDB-Rare, which were −0.010, 0.032, and 0.007.

Figure 6C shows that the correlation coefficients between perplexity and top-50 accuracy in GMDB-Frequent were 0.053, 0.002, and 0.034. Figure 6D shows the corresponding coefficients in GMDB-Rare, which were −0.027, −0.016, and −0.016. All correlation coefficients had absolute values below 0.10 and all Holm-adjusted P>.05, indicating weak associations of self-reported confidence and perplexity with diagnostic hits.

Figure S3 in Multimedia Appendix 1 presents top-50 accuracy stratified by the models’ assessments of evidence sufficiency. Under the raw image condition, all 3 models predominantly reported “insufficient evidence,” and the direction of between-group accuracy differences varied by model and subset. Detailed statistical results are provided in Multimedia Appendix 2.

‎

Prediction Overlap and Composition

The top-50 candidate diagnosis sets generated by each model showed substantial overlap across input conditions (Figure 7A for GMDB-Frequent and Figure 7B for GMDB-Rare). Gemma-4-31B had the highest mean pairwise Jaccard index, approximately 0.52 in both subsets; the corresponding values were approximately 0.27 to 0.31 for Qwen3.6-35B-A3B and 0.31 to 0.36 for Qwen3.8-27B. The overlap between candidate sets under the red-circle and mismatched red-circle conditions was similar to that observed across different processing conditions for the same image. This finding suggests limited changes in candidate sets when the patient image was replaced.

Top-1 predictions from all 3 models were highly concentrated in a small number of diseases. In GMDB-Frequent (Figure 7C), the 5 most frequent diseases in the ground-truth labels accounted for 16.8% (148/882) of all queries. Under the red-circle condition, the 5 most frequently predicted diseases accounted for 92.5% (816/882), 81.4% (718/882), and 72% (635/882) of predictions from Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B, respectively. In GMDB-Rare (Figure 7D), the corresponding proportion for the ground-truth labels was 4.2% (162/3864), while the proportions for the 3 models reached 94.7% (3658/3864), 85% (3283/3864), and 71% (2740/3864), respectively.

Gemma-4-31B’s predictions were primarily concentrated in Williams syndrome and Cornelia de Lange syndrome, while Qwen3.6-35B-A3B’s predictions were primarily concentrated in Kabuki syndrome and Rubinstein-Taybi syndrome. Qwen3.8-27B frequently predicted Kabuki syndrome, fetal alcohol spectrum disorder, and Coffin-Siris syndrome 1. Similar concentration patterns were observed under the mismatched red-circle condition and the 2-stage image-to-phenotype baseline.

‎

Discussion

Principal Findings

This study evaluated the ability of 3 locally deployed MLLMs to perform diagnosis based on facial images and examined the effects of image enhancement, red-circle prompting, and the 2-stage image-to-phenotype baseline. Overall diagnostic accuracy was limited, with particularly low performance on GMDB-Rare. Compared with the raw image condition, red-circle prompting produced numerical increases across most model-subset combinations, with a statistically significant improvement observed for Qwen3.8-27B on GMDB-Rare. The quality-stratified analysis further showed that Gemma-4-31B benefited from red-circle prompting in the good subgroup of GMDB-Frequent. The 2-stage image-to-phenotype baseline also improved accuracy for specific models and subsets. Matching red circles yielded numerical accuracy increases relative to mismatched red circles across all comparisons, although statistical support for these differences was limited. Together with the overlap among prediction sets after patient-image replacement and the concentration of top-1 outputs, these findings suggest that model outputs were only weakly constrained by patient-specific facial information. Improvements associated with different input strategies were concentrated in specific models and disease subsets, and their applicability requires further validation. In addition, overall accuracy varied only slightly across repeated inferences, while self-reported confidence and diagnostic-list perplexity showed weak associations with diagnostic accuracy.

Diagnostic Utility of Visual and Phenotype-Based Prompting

Comparisons between the red-circle and raw image conditions reflect the combined effects of image enhancement and circle annotation. Comparisons between the red-circle and super-resolution conditions provide a more direct assessment of regional prompting. Gemma-4-31B showed a significant benefit in the good subgroup of GMDB-Frequent, suggesting that clinically relevant regional localization may help the model use facial cues. The higher accuracy observed in the good subgroup may also relate to case composition and phenotype recognizability. In the expert-annotated circle experiment, the direction of the difference between the 2 circle sources varied across models, with limited statistical support. These findings suggest that circle quality and the model’s ability to use local features for disease discrimination should be evaluated together.

The 2-stage image-to-phenotype baseline converts visual cues into explicit phenotype descriptions, which may help the model organize features and retrieve relevant disease knowledge. Its significant improvements relative to the raw image condition were concentrated in specific models and subsets. Comparisons with the red-circle condition yielded Holm-adjusted P>.05 across all settings, and the additional diagnostic value of phenotype text requires further validation.

Stability and Uncertainty

Overall accuracy varied only slightly across the 3 repeated-inference runs, indicating a degree of run-to-run stability in diagnostic performance on the selected queries. This finding reflects variation in the overall hit rate across runs; prediction consistency for individual queries and the stability of the benefits of visual prompting require further evaluation. GMDB-Rare had a low number of diagnostic hits, so changes in the outcomes of a small number of queries could affect accuracy. Its stability should therefore be interpreted in the context of sample size.

Self-reported confidence and perplexity showed weak associations with diagnostic hits, and the direction of accuracy differences between evidence-sufficiency groups varied across models and subsets. These findings suggest that these metrics provide limited information about diagnostic reliability in the current task. Self-reported confidence reflects the model’s subjective estimate that the candidate list contains the correct diagnosis, while perplexity reflects the token probabilities of the generated sequence of disease names. The use of self-reported confidence, perplexity, or evidence-sufficiency judgments to identify diagnostic outputs with low reliability therefore requires further validation [31].

Limitations

This study has several limitations. First, there is a potential risk of training-data contamination [32,33]: publicly available facial images from the evaluated cases, such as patient photographs published in case reports, may have been included in model training data, potentially affecting estimates of diagnostic accuracy and assessments of generalizability. Limited disclosure of case-level sources in model training data makes this overlap difficult to quantify directly. Second, clinical comparisons were limited to physician ratings of circle quality and a comparison between expert-annotated and automatically generated circles for 100 queries. These analyses primarily assessed the clinical relevance of the annotations and their impact on model diagnostic performance. Direct comparisons with physician diagnoses and the clinical utility of the models as assistive tools require further evaluation. Third, real-world population prevalence data for individual diseases are limited, and a disease-frequency baseline remains to be added. In preliminary image-free controls, all models declined to answer because images were unavailable. The respective contributions of disease priors and visual information to diagnostic ranking require further clarification. Fourth, repeated inference covered only 200 queries and 3 runs, reflecting run-to-run variability within the selected subset. The small number of diagnostic hits in GMDB-Rare also made the results sensitive to changes in the outcomes of a small number of queries. Fifth, simulated MONDO ID substitutions provide an indirect assessment of sensitivity to mapping errors. However, they may differ from actual mapping errors, so this analysis cannot quantify the effect of actual mapping errors on diagnostic accuracy. Sixth, diagnostic accuracy depends on the normalization of disease names to MONDO. Manual verification of 500 unique disease names found that 486 (97.2%) mappings were correct. Residual mapping errors may affect accuracy estimates and comparisons across conditions. Seventh, the disease spectrum, age distribution, ancestry, and image acquisition conditions of the disease-labeled facial images in GMDB may differ from those encountered in routine outpatient practice. The effects of these differences on model performance remain unquantified in this study. Previous studies on facial image–assisted diagnosis have also highlighted the influence of image quality and population coverage on system performance [34]. Eighth, the information used in this study was limited to facial images and model-generated phenotype descriptions. Performance incorporating comprehensive physician-verified phenotypes, developmental history, family history, and genetic testing information requires further evaluation. Ninth, the absence of query-level inference runtime records limited the assessment of runtime performance.

Future Directions

Future research could compare visual prompts based on points, bounding boxes, and masks, as well as in-context examples, to evaluate the effects of different guidance strategies. Facial landmark detection could provide standardized geometric localization, while disease-related phenotype annotations could support the training and validation of abnormality-recognition modules to generate regional prompts and assess their localization quality and diagnostic utility. For information integration, facial analysis could be combined with human phenotype ontology terms [35], family history, and genetic testing results [36,37]. Multimodal retrieval-augmented generation could further incorporate relevant cases and disease knowledge [38-40] to evaluate the contributions of more comprehensive patient information and external evidence to diagnosis.

To investigate training-data exposure, future research could explore the number of disease-related publications in PubMed as a proxy for literature exposure and analyze its association with top-50 accuracy. Manual verification of MONDO mappings could also be expanded to identify disease names prone to mapping errors and common error patterns, guide improvements to the mapping tool, and assess the impact of mapping corrections on diagnostic accuracy. To assess run-to-run stability, the number of queries and repeated-inference runs could be increased to further quantify fluctuations in overall accuracy across runs. In addition, the relationships of self-reported confidence, perplexity, and evidence sufficiency with disease frequency, image quality, and prompt quality should be analyzed to evaluate the ability of these metrics to identify diagnostic outputs with low reliability.

Clinical validation should compare diagnoses made by physicians and models under the same information conditions and evaluate accuracy, diagnostic time, and error types for physicians making independent and model-assisted judgments. Ideally, further studies should incorporate prospective cases from multiple centers and conduct stratified analyses by age, ethnicity, and disease spectrum to clarify the scope of clinical applicability and model performance across different populations.

Conclusions

Under the experimental settings evaluated in this study, Gemma-4-31B, Qwen3.6-35B-A3B, and Qwen3.8-27B showed limited ability to diagnose rare genetic diseases using facial images alone. Performance was particularly low on GMDB-Rare. The benefits of image enhancement, red-circle prompting, and the 2-stage image-to-phenotype workflow were primarily observed in specific models and disease subsets. Together, the mismatched-image controls, prediction-set overlap, and disease-composition analyses suggest that model outputs were only weakly constrained by patient-specific facial information. Future research should strengthen the integration of patient phenotypes with disease knowledge and validate the models’ clinical utility as assistive tools through direct comparisons with physicians and multicenter prospective studies.