Participants were recruited via convenience sampling from community care centers in Taiwan. A member of the research team visited these centers, provided an oral presentation to groups of older adults explaining the study’s purpose and procedures, and invited interested individuals to enroll. Participants were eligible if they met the following criteria: (1) aged 65 years or older, (2) having had no hospitalization in the past 6 months, and (3) willing to participate in the study. Exclusion criteria were a doctor-diagnosed disability or an unwillingness to record videos during the CDVT assessment.

This study consisted of 2 waves of data collection. In the first wave, participants were assessed once to gather cross-sectional data. Assessments included the CDVT and Montreal Cognitive Assessment (MoCA) [21]. In the second wave, participants were assessed twice, with a 2-week interval between assessments, to collect test-retest data. Both assessments involved the CDVT and MoCA. Additionally, in the first assessment, the Stroop Color Word Test (SCW) [22] and both parts of the Color Trails Test (CTT) [23] were assessed. At the start of the first assessment session for each participant, a trained assessor administered a brief questionnaire to collect demographic information, including age, sex, and educational attainment (Multimedia Appendix 1). Each participant was individually assessed by a trained assessor in a quiet, one-on-one setting at a community care center. All assessments were conducted on a laptop computer with a 15.6-inch screen, an Intel i5 processor, 8 GB of DDR4 RAM, and a GTX 950 graphics card. Participants were seated approximately 50 cm from the screen. Video recordings capturing participants’ heads and faces were obtained using the laptop’s built-in HD webcam (1.3 megapixels) during CDVT assessments.

To determine whether the AI-CDVT can evaluate participants’ attention according to their facial videos and CDVT output, we developed the AI-CDVT in 3 steps, retrieving attentional features, establishing an AI-based scoring model, and validating the AI-CDVT. In the first step, we adopted OpenFace (CMU MultiComp Lab) [28] to retrieve attentional features from a participant’s face in videos. Attentional features were extracted from each video frame. The eight primary features included: (1-2) the distance between the eyelids for each eye, serving as a continuous measure of eye openness (a smaller distance indicates greater closure, as a proxy for blinks); (3) the distance between the lips, indicating mouth opening or yawning; (4‐6) head rotation angles across 3 axes, corresponding to pitch (nodding), yaw (side-to-side rotation), and roll (ear-to-shoulder tilt); and (7-8) the X-Y coordinates of the estimated gaze point. For each of the 120 trials in the CDVT, we aggregated the frame-by-frame data by calculating the mean value for each of these 8 features. Within any given trial, video frames where OpenFace failed to successfully track facial features were excluded from this calculation; the mean was computed only from the successfully captured frames. Furthermore, because the CDVT is a computerized test that requires a response before proceeding to the next item, there were no missing test items. This process yielded a single value per feature for each trial. Finally, we calculated the overall mean and SD of these 120 trial-level values for each feature. These 16 summary statistics (8 features×2 statistics), along with the mean and SD of the reaction time and response accuracy from the original CDVT, constituted the final set of 20 input features for the AI model. All input features were normalized to a common scale before being used in the model.

In the second step, we adopted an AI algorithm, the Extreme Gradient Boosting (XGBoost) regressor [29], to receive inputs of attentional features and estimate scores of the CDVT as scores of the AI-CDVT. The XGBoost regressor leverages the aggregation of random forest regressors to improve the predictive accuracy and has been successfully used in medication situations [29]. In addition, the XGBoost regressor offers an importance ranking of attentional features to generate AI-CDVT scores.

In the third step, we validated the AI-CDVT using a 3-fold cross-validation procedure [30]. In this procedure, we randomly separated our data into 3 subdatasets. Then, we validated the AI-CDVT 3 times. For each validation instance, we used 2 subdatasets to train the XGBoost regressor and used the remaining subdataset to validate the convergent validity and test-retest reliability. Notably, a different subdataset was used each time for validation.

Then, we calculated the mean and SD of the indices of validity across the 3 validation times. The convergent validity was examined using Pearson r values to evaluate associations between AI-CDVT scores and those of the MoCA, CTT, and SCW. In addition, we also calculated Pearson r values between CDVT scores and those of the MoCA, CTT, and SCW. If Pearson r values of the AI-CDVT were similar to those of the CDVT, the convergent validity of the AI-CDV was considered good [31].

The test-retest reliability was examined using an intraclass correlation coefficient (ICC) to evaluate the agreement of scores in the test and retest assessment sections. An ICC value of > 0.75 indicates good test-retest reliability [32].

Finally, to provide a more rigorous evaluation of the AI-CDVT’s clinical utility beyond correlational metrics, we conducted a supplementary classification analysis. The goal was to determine whether the multimodal features captured by the AI-CDVT offered incremental value in predicting a clinically relevant outcome. We used the MoCA score as a proxy for the cognitive status, dichotomizing participants into 2 groups based on the common clinical cutoff for potential cognitive impairment “at risk” (MoCA score ≤25) and “not at risk” (MoCA score >25). We then compared the performance of the 2 XGBoost classification models in predicting this binary outcome. The first model (“CDVT-Only”) used only the mean reaction time and mean accuracy from the CDVT as input features. The second model (“AI-CDVT Top 5”) used the 5 most important features identified in our initial regression model (see Table 1), which included a mix of performance and facial metrics. Specifically, these top 5 features were the mean reaction time, the mean distance between the right eyelids, the mean Y coordinate of the gaze point, the mean accuracy, and the mean angle of head rotation in the Y axis (yaw). Model performance was evaluated using the accuracy and F₁-score across the same 3-fold cross-validation procedure.

This study was approved by an institutional review board (Taipei Medical University, approval number: N202010008), and all participants provided written informed consent. Participants received a small honorarium of a gift voucher equivalent to US $3 for their time and participation. The informed consent process explicitly detailed the nature of the video recording and its purpose. To ensure data privacy and confidentiality, all video data were stored on an encrypted, offline hard drive. The raw video files were permanently deleted immediately after the deidentified facial features were extracted by OpenFace software, and only these numerical, nonidentifiable data points were retained for analysis.

The primary rationale for developing the AI-CDVT was to determine if integrating multiple, objective behavioral data streams could enhance the psychometric robustness of a standard sustained-attention test. While the AI-CDVT did not shorten the test’s administration time nor demonstrate superior correlations with external cognitive measures, our findings support the study’s main hypothesis: the AI-CDVT achieved a notable improvement in the test-retest reliability while maintaining convergent validity comparable to the original CDVT. This suggests that by capturing a richer, multimodal snapshot of an individual’s attentional state, the AI-CDVT offers a more stable and reliable assessment tool.

Compared to the CDVT, the AI-CDVT demonstrated equivalent validity and improved test-retest reliability. This enhanced reliability might be attributed to the incorporation of additional attentional features in the AI-CDVT [33,34]. By incorporating a broader range of attentional features, AI-CDVT scores likely benefited from greater stability and robustness. The AI-CDVT score may be less vulnerable to fluctuations solely due to response speed or accuracy. The findings suggest that incorporating a wider range of attentional features during assessments can lead to more reliable scores.

While the AI-CDVT’s convergent validity correlations were similar to those of the original CDVT, its primary clinical and functional value lies not in superior predictive accuracy but in its enhanced psychometric robustness and the potential for a more nuanced interpretation of attentional performance. The improved test-retest reliability, for example, is a direct clinical benefit, as it provides a more stable and trustworthy score for longitudinal monitoring or evaluating intervention effects [32]. Furthermore, by capturing a wider array of behaviors like gaze shifts and eyelid closure, the AI-CDVT provides a richer dataset. This could allow clinicians to move beyond a single performance score to understand the underlying nature of an individual’s attentional difficulties, distinguishing, for instance, between general cognitive slowing and specific lapses in vigilance [33]. This multimodal approach offers a more holistic view of sustained attention [34], paving the way for future research into distinct behavioral phenotypes of attentional decline that are invisible to traditional, response-time-based measures.

The results of our supplementary classification analysis warrant careful interpretation. The modest improvements (1.4% in accuracy and 2.0% in F₁-score), while not dramatic, provide a crucial proof-of-concept. They suggest that the multimodal features captured by the AI-CDVT contain a small but detectable signal that is relevant to clinical outcomes (ie, potential cognitive impairment as flagged by the MoCA). This finding lends support to our central hypothesis that integrating objective behavioral markers, even if their individual predictive power is small, can incrementally enhance the clinical utility of a traditional cognitive test. While the immediate clinical impact of this gain is limited, it establishes a methodological foundation for future work. It is plausible that this incremental value could be magnified in larger, more clinically diverse samples or by using more advanced machine learning architectures, highlighting a promising avenue for subsequent research.

It is important to appropriately position the contribution of this study. We did not develop a new AI algorithm from the ground up; rather, our innovation lies in the practical application and integration of established, open-source tools (OpenFace and XGBoost) to enhance a standard clinical assessment. The novelty of this work is therefore not in a deep methodological invention but in demonstrating the feasibility and clinical utility of creating a multimodal assessment of sustained attention. By showing that combining behavioral response data with easily captured facial metrics can improve psychometric properties like reliability, this study provides a proof-of-concept and a methodological template for other researchers aiming to enrich traditional neuropsychological tests with objective, behavioral data streams.

By leveraging the flexibility of AI models, such as XGBoost regressors, the AI-CDVT can potentially address the challenge of interpreting individual attentional features in isolation [35]. For instance, directly comparing attention between individuals with high accuracy but slow response times and those with lower accuracy but faster responses can be difficult [36]. The AI-CDVT illustrates a viable method for integrating and interpreting various attentional features into a unified score for future studies and attentional tests.

A related methodological consideration is the selection of the 20 input features. This set was not chosen arbitrarily. Each feature was included because it had been identified in the existing scientific literature as a physiological or behavioral marker related to attention and vigilance [11,12,37]. Our approach was to build a comprehensive model based on these established, theory-driven features. While this could introduce redundancy between some inputs (eg, left and right eyelid distance), the chosen XGBoost algorithm is robust to such multicollinearity, as it inherently performs feature selection during its training process [38]. The final feature importance rankings in Table 1 are a direct result of this process, demonstrating how the model itself identified the most valuable contributors from the initial set of theory-driven features.

An examination of the feature importance rankings (Table 1) provides insights into how the AI-CDVT achieves its robust performance. While traditional metrics like reaction time and accuracy are unsurprisingly the most critical predictors, the model also heavily weighs physiological and behavioral markers. For example, the “distance between the eyelids” emerged as a top feature. This is consistent with literature linking decreased eyelid aperture and blink rate dynamics to drowsiness and lapses in vigilance [39]. Similarly, the “Y coordinate of the gaze point” was highly predictive, likely because a downward shift in gaze is a well-established behavioral marker of task disengagement and mind-wandering [40,41]. The inclusion of these features allows the AI-CDVT to capture subtle, moment-to-moment fluctuations in attentional states that are not reflected in response times alone, thereby providing a more comprehensive and ecologically valid assessment.

A final consideration is the interpretability of the AI model, which is a critical factor for its clinical adoption [42,43]. Complex models like XGBoost are often termed “black boxes” because they do not produce a simple, transparent formula in the way a linear regression model can [44]. There is an inherent tradeoff between the high predictive accuracy of such models and their direct interpretability [45]. In this study, our primary tool for interpretation is the feature importance table (Table 1). While it does not explain how the features are combined for any single individual, it provides clinicians with a clear and valuable understanding of what the model is paying attention to [46]. It confirms that the AI-CDVT’s score is driven by a combination of performance metrics and behavioral patterns (eg, eye closure and gaze aversion) that are clinically consistent with inattention. This allows clinicians to trust that the model’s logic aligns with established knowledge, even if the precise weighting algorithm remains complex [47,48].

This study has several notable limitations. First, the generalizability of our findings is constrained by the characteristics of our sample. The study included cognitively healthy older adults with MoCA scores in a narrow range, limiting the applicability of the results to clinical populations with cognitive impairments. Furthermore, the sample was predominantly female (64/87, 73.6%), raising concerns about potential gender bias in the AI model’s performance [49]. The test-retest reliability analysis was also based on a relatively small subset (n=66), which warrants caution in the interpretation of ICC values until replicated with a larger sample.

Second, the scope of our validation methodology was limited. The model’s performance was evaluated using internal cross-validation; the lack of an external validation with an independent dataset means that the model’s generalizability remains to be confirmed. Our psychometric evaluation also focused primarily on correlational and reliability metrics. Other key psychometric properties, such as responsiveness to change, were also not assessed.

Finally, it is important to acknowledge a conceptual limitation regarding what is being measured. The facial and behavioral metrics captured by the AI-CDVT, such as eye closure, are well-established proxies for vigilance. However, they are not direct measures of underlying cognitive processes. Therefore, the observed correlations with cognitive tests should be interpreted as a relationship between observable behaviors and cognitive performance, not as evidence that the AI-CDVT measures cognition in the same way as traditional tests.

Building on these findings, future research should prioritize several key areas. First, the AI-CDVT must be validated in a larger, more diverse cohort, specifically including a more balanced gender representation and a wider range of cognitive abilities, to address the limitations of our current sample. Second, external validation on an independent dataset is essential to confirm the model’s generalizability. Third, its utility in clinical populations, such as individuals with mild cognitive impairment or dementia, should be explored to determine its diagnostic and monitoring potential. Finally, longitudinal studies are needed to assess other key psychometric properties, such as responsiveness to change over time or in response to an intervention.

The objective of this study was to address a key psychometric weakness in a standard test of sustained attention by transforming it from a unimodal to a multimodal assessment. We demonstrated that using accessible AI tools to integrate objective behavioral data with traditional performance metrics, and we developed an AI-CDVT with superior test-retest reliability compared to the original version, while maintaining its convergent validity. The primary contribution of this work is not the development of a novel algorithm, but the demonstration that the psychometric robustness of established clinical tools can significantly be enhanced through this multimodal approach. The resulting AI-CDVT represents a more stable and reliable instrument for assessing sustained attention in older adults.