The study prospectively validated three DR detection AI algorithms (validation phase) and assessed the technical feasibility of implementing a validated AI algorithm in public health settings (implementation phase). The STARD (Standards for Reporting of Diagnostic Accuracy Studies) checklist [19] was used to report the completeness and transparency of the diagnostic accuracy study and the iCHECK-DH framework to enhance the completeness and transparency of reporting related to the digital health implementation components [20]. Figure 1 summarizes the study design.

The study received approval from the Postgraduate Institute of Medical Education and Research (PGIMER) Institutional Ethics Committee (PGI/IEC/2020/001342) and followed the recommendations of the Declaration of Helsinki. The study was prospectively registered with the Clinical Trials Registry India (CTRI/2022/10/046185). Individuals aged > 30 years with a history of diabetes mellitus were screened for DR, following the National Program for Prevention and Control of Non-Communicable Diseases [21]. Informed consent was obtained, and their routine care remained unchanged. All data used in this study were anonymized prior to analysis, with no personal identifiers retained.

Validation was conducted from March to June 2021 at the Department of Endocrinology, PGIMER, Chandigarh, and a Primary Health Center (PHC) in Khizrabad, District Mohali, Punjab. The AI-enabled DRS was implemented between February 2022 and June 2022 at a community health center (CHC) in Badhani Kalan, Moga District, Punjab, India.

Sample sizes (validation=256; implementation=348) were calculated assuming DR prevalence of 17%, nongradable image rates of 18.4% in validation and 30.3% during implementation, 70% sensitivity, 86% specificity, 95% CI, and nonresponse rates of 10% and 15% in validation and implementation, respectively [3,22-25].

Based on a scoping review, five AI companies that used cloud-based AI (four Indian, one international) were invited. Before validation, they received the study objectives and camera or image specifications.

The study data was collected using Research Electronic Data Capture (REDCap) [33]. Deidentified data were downloaded from REDCap and imported into Stata/IC (version 15.1; StataCorp) [34] for analysis.

The AI platform’s sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were estimated with 95% exact binomial CIs. A P value of <·05 was considered significant for all statistical tests. A κ value was calculated monthly during the implementation phase to assess the optometrists learning in image acquisition and the AI algorithm’s performance in image quality and DR diagnosis. This measured the agreement between AI and RSs for image quality and DR diagnosis. The folders with missing bilateral or macula-centered images and people without diabetes mellitus were excluded from the analysis. All analyses were conducted on an eye-wise basis.

Among the 250 participants in the validation phase, 182 were recruited at the PHC Khijrabad and 68 from the endocrinology clinic. The mean age of the participants was 53.5 (SD 7) years in the PHC and 47.2 (SD 4) years in the Endocrinology clinic, respectively. Overall, 87/250 (48.3%) participants in the PHC group were men and 95 (51.7%) were women, and those in the Endocrinology group included 38 (55.8%) men and 30 (44.1%) women. In the implementation phase, 343 participants were recruited. The participants’ mean age was 58.48 (SD 10.41) years, with 202/343 (59%) women and 140 (41%) men (Table 1).

Figure 4 shows κ variations for image quality and DR grades over 4.5 months when the health system optometrist captured the images. κ values of image quality increased from 0 in February to 0.74 in June, and for DR grade from 0 to 0.71 in June. This steady improvement could be attributed to the enhanced image quality captured by the optometrist over the study period and higher; in DR, this increase can be linked to better image quality and higher detection, with slight dips due to variation in DR severity or case complexity.

Of 64 referred participants, 28 (43.8%) were contacted; only 9 (14%) adhered to referral advice and visited an ophthalmologist for a review. Of these, 1 received an Optical Coherence Tomography referral, 5 went to facilities without eye care services, 1 received eye drops, 1 was advised antivascular endothelial growth factor injection, 1 left after receiving laser treatment, and 1 had a follow-up visit.

Reasons for nonadherence included harvesting season (n=11), lack of family support (n=2), time constraints (n=2), extended absence from home (n=1), financial dependence on family (n=2), and comorbidities (n=1).

This study is among the first in India to validate multiple AI algorithms for DRS, critically assessing their technical feasibility for integration into real-world public health settings. We evaluated the best-performing AI algorithm using images captured by a skilled research optometrist. This study outlines the Government of India’s efforts to foster an ecosystem that ensures the integration of responsible AI technologies before application to end users [35].

In our study, AI system performance during validation varied significantly: sensitivity (59.7%97.74%), specificity (14.25%96.01%), PPV (30.16%86.67%), NPV (85%94.34%), and accuracy (37.19%88.43%). A multicenter study by Lee et al systematically compared seven AI-based DRS algorithms, revealing high NPVs (82.72%93.69%) but widely varying sensitivities (50.98%85.90%) in real-world performance [36]. In pivotal clinical trials for the IDX AI algorithm, the FDA’s benchmark for superiority was set at 85% sensitivity and 82.5% specificity [13]. Although AI-3 sensitivity (68.3%) was below the FDA’s >85% threshold, it outperformed other AIs in other diagnostic metrics: specificity (96.01%), PPV (86.67%), NPV (88.92%), and accuracy (88.43%). Prospective validation of a DRS algorithm at two Indian tertiary eye care hospitals demonstrated 89% sensitivity and 92% specificity on nonmydriatic images [37].

However, there are challenges in comparing algorithms using published results due to the variations in study methodologies [38]. One challenge is the dependence of an AI algorithm’s accuracy on the quality of the retinal images obtained [27]. Our study included ungradable images in validation, contrasting with many studies that preprocess or exclude lower-quality images in their training data sets [12,13,39-41] (Figure S6 in Multimedia Appendix 1). Excluding these images fails to reflect real-world settings, potentially lowering the algorithm’s performance [13,41]. To date, validation studies demonstrate that most AI algorithms using mydriatic fundus images achieve high diagnostic accuracy [42]. In the validation phase, the dark room environment facilitated nonmydriatic conditions, yielding 0% ungradability for AI-1 and AI-3, 6.2% for AI-2, and 3.2% by human grader, contrasting with 1830% in LMIC settings per a systematic review [38,42].

Cataracts are a leading cause of ungradable images [28]. However, no images with cataract or other media opacities were classified as ungradable during implementation, potentially affecting AI-3 specificity (Table 2). Notably, the AI-3 algorithm failed to detect 21 (42%) eyes with cataracts, categorized as gradable. The AI-3 algorithm’s sensitivity increased significantly from 68.4% to 99.6%, while specificity decreased from 96% to 64.7% between the validation and implementation phases. The sensitivity likely improved post validation due to algorithm training (Table S2 in Multimedia Appendix 1). The presence of cataracts, affecting media opacity and gradability, likely could have decreased specificity by increasing false positives [43]. After algorithmic training, AI-3 detected DME (Table 2) and sensitivity (26.5%), highlighting the need for improved training to enhance DME sensitivity and subsequently achieve higher RDR sensitivity (78.9%).

Optometrists serve as frontline eye care providers globally and are ideally positioned for task-sharing in DRS. Their integration into DRS and care pathways is well-established in diverse models worldwide, supporting sustainable and scalable eye care delivery [44-46]. In this study, a health system optometrist was trained and engaged in DRS at the CHC, with oversight from the research team. Over the study period, image quality and DR detection improved, reflecting a positive learning curve (Figure 3). These findings highlight the potential of optometrist-led AI-assisted screening in strengthening task-shifting models and expanding access to DR care in resource-limited settings [47-49]. Implementing an opportunistic AI-enabled DRS model holds promise for enhancing detection rates [47]. However, our study’s adherence to referral recommendations remains suboptimal, with approximately 14% of participants attending recommended follow-up visits at an eye care facility. Key barriers to referral adherence include awareness gaps, logistical challenges (ie, travel, DR related cost), and persisting health system limitations, including weak referral pathways and poor patient tracking [48]. Personalized approaches, such as phone calls, voicemails, and detailed result letters, are shown to be effective in improving referral adherence [49]. Low referral adherence underscores the need for effective, coordinated referral pathways before introducing new screening models, while recognizing the vital role of teleophthalmology in such systems [47,50].

A notable strength of this study is its real-time implementation, conducted during regular clinics by a health system optometrist within a public health care setting. This marks a significant milestone that is likely to substantially enhance DRS services. The study showed that training and monitoring fundus image quality could significantly improve the effectiveness of the AI-enabled DRS program.

However, several limitations should be considered when interpreting the results. Using a single fundus camera may limit generalizability, as AI performance may vary across camera models and imaging conditions. Additionally, the study relied on a specific AI algorithm, which may not account for variations in other models or evolving AI systems. Setting-specific constraints such as workflow integration and infrastructure availability could also impact scalability. Furthermore, potential biases in real-world data, such as variations in patient demographics, image quality, and disease prevalence, may influence AI performance and limit broader applicability. Integrating DRS was feasible; however, we could monitor the optometrist only for three months poststudy. Regular monitoring is crucial for program sustainability and healthcare provider motivation.

Operational challenges in DRS included limited patient access due to long travel distances, poor transport, and low awareness; inadequate infrastructure; and ergonomic barriers affecting both patients and screeners. Uncontrolled lighting and power issues led to 26% ungradable images. Adaptive measures, such as transport support, ergonomic adjustments, darkroom setups, and equipment reinforcements, raised image gradeability to 95.6% and improved efficiency. Addressing these barriers through infrastructure upgrades, controlled environments, and community facilitation is essential for sustainable DRS in primary health care systems.

In conclusion, this study highlights the essential role of systematic AI validation in integrating technology responsibly into clinical workflows. By demonstrating AI’s feasibility for DRS in Indian public health settings, our findings support scalable solutions to improve health care accessibility in resource-constrained contexts across the Global South. However, long-term sustainability and large-scale implementation will require ongoing funding, a robust workforce, and effective policy integration. Further research is needed to evaluate the large-scale deployment of AI-driven screening strategies, examining their clinical effectiveness, cost-effectiveness, and the challenges of implementing them in real-world settings.