Stroke is currently considered one of the top global causes of disability, with most survivors of stroke in need of inpatient rehabilitation at specialized clinical centers [1]. Recent studies have reported that almost 40% of survivors of stroke sustain cognitive impairment [2]. The World Health Organization definition of cognitive impairment has been recently referred [3] to as problems experienced by an individual in remembering things, making decisions, learning abilities or concentrating on tasks that affect their everyday life.

Cognitive rehabilitation (neuropsychological rehabilitation) relies on brain plasticity to induce neuroplastic changes to compensate for cognitive impairments [4]. Brain injury is one of the key causes of cognitive impairment; however, other factors contribute to the ever-increasing number of people in need of cognitive rehabilitation (one of them being the global trend in population aging).

One of the most frequent cognitive problems reported by poststroke patients in their daily lives is related to memory loss [5,6]. To date, associations between factors for ischemic stroke and clinical outcomes have been analyzed predominantly in older rather than younger patients [7]; however, the incidence rates of ischemic stroke have increased in young adults in the United States [8] and also in Europe [9].

New strategies for providing cognitive rehabilitation services are constantly required and are continuously being integrated into clinical practice [10]. One such strategy is the use of web-based systems, and several of these systems have already been used to optimize cognitive interventions [11,12]. However, because of the relatively recent development of these services, the best strategies to integrate them into everyday clinical practice are still unclear [13]. Nevertheless, strategies targeting the personalization of the proposed activities for patients according to their specific needs appear to be more effective [14].

A typical cognitive rehabilitation program mainly provides exercises that require repetitive use of the impaired cognitive system in a progressively more demanding [15] sequence of tasks. The impact of a task or exercise execution depends on the ratio between the skills of the treated patient and the challenges involved in the execution of the task itself. Thus, determining the correct training schedule requires a quite precise trade-off between sufficient stimulation and sufficiently achievable tasks, which is far from trivial and is still an open issue, both empirically and theoretically [16,17].

Furthermore, prediction of specific outcomes after stroke rehabilitation is used by clinicians to improve the accuracy of prognoses, set attainable goals, reach shared decisions, personalize rehabilitation plans, and inform patients and relatives [18].

In this study, we propose the application of Elo ratings to provide clinicians with a ranking of patients at an early stage of cognitive rehabilitation by using the results of web-based cognitive rehabilitation tasks. We hypothesize that (1) such ranking of patients will allow clinicians to match patient’s skills with task difficulties, thereby enabling better treatment personalization, and (2) such a rating will be a significant predictor of patients’ outcomes for memory cognitive function. The original proposal of the Elo rating system was designed to rate chess players, and the rating system was named after its creator Arpad Elo [19].

The Elo system works as follows: an initial rating is assigned to each player every time a player plays a match. This rating is updated for both players depending on the result of the match. If the winner is the player with the higher rating, the update is small, and it is larger depending on how unexpected the victory is, according to their previous ratings [20].

The basic Elo rating system is used in several types of contests beyond chess, for example, football [21]; however, different applications have been extensively reported elsewhere. It has been used for eliciting user preferences in community-based sites [22], assessing security and vulnerability risks [23], ranking posts in web-based forums [24], rating patterns in videogames [25], detecting fabric defects in the textile industry [26], providing students with individualized learning materials in educational settings [20], studying traffic congestion in urban transportation [27], studying dominance hierarchies in behavioral and evolutionary animal ecology [28], forecasting sales and optimizing prices of new product releases [29], allocating resources for criminal justice to support supervision officers [30], and identifying people using facial comparative descriptions [31].

Nevertheless, to the best of our knowledge, Elo ratings have not been applied in cognitive rehabilitation in general or in the specific use–case of a web-based application where patients perform web-based cognitive tasks during their rehabilitation period.

In this study, we propose that instead of considering matches between, for example, chess players, we consider matches between patients and web-based cognitive rehabilitation tasks.

The aims of this study are (1) to demonstrate the feasibility of the approach by presenting a synthetic data set where we obtain an Elo rating for each patient by considering each execution of a cognitive rehabilitation task by the patient as a match between the patient and the task; (2) to obtain the Elo rating of each patient in a real rehabilitation setting where adult patients with ischemic stroke follow cognitive rehabilitation by executing web-based rehabilitation tasks and use these Elo ratings to perform a stratification of patients into 3 groups according to their Elo rating (low, middle, and high); (3) to analyze the relationship among the three Elo rating levels and the proportion of tasks executed at three increasing difficulty levels (1, 2, and 3) with the rehabilitation outcomes in the memory cognitive function; and (4) to develop and internally validate four predictive models for auditory verbal learning memory and working memory outcomes using Elo ratings obtained at early stages of rehabilitation as independent variables and state-of-the-art variables (eg, sex, age, and length of stay). The first two models are developed for predicting auditory verbal learning memory and working memory at discharge and the other two for predicting improvements in auditory verbal learning memory and working memory at discharge.

The setting was the inpatient acquired brain injury rehabilitation unit of the Institut Guttmann hospital, a specialized clinical center certified in quality of care and patient safety (Joint Commission International since 2005 and consecutively recertified in 2009, 2012, and 2018). The initial study population consisted of 344 patients with ischemic stroke who were consecutively admitted for inpatient rehabilitation from March 2009 to September 2019. Patients were included in the study if they had been admitted within 180 days of the onset of an ischemic stroke. Patients who were admitted >180 days after a stroke (31/344, 9%), who had no cognitive assessment within a week after stroke rehabilitation admission (18/344, 5.2%), or had missing data (7/344, 2%) were excluded. Therefore, 83.7% (288/344) of the patients were available for analysis. Patients with aphasia were not included in the n=344 initial sample as they follow a different rehabilitation protocol involving a different set of cognitive assessments and, therefore, need to be analyzed separately (in future work).

At admission, each patient was assigned a physician who coordinated the rehabilitation team (a nurse, a neuropsychologist, a physiotherapist, an occupational therapist, a social worker, and a clinical psychologist based on the characteristics of the case). Therefore, admission and discharge cognitive assessments (as well as all clinical and demographic data analyzed in this study) were systematically recorded in the electronic health records of the hospital. The authors confirm that this study is compliant with the Helsinki Declaration of 1975, as revised in 2008, and it was approved by the Ethics Committee of Clinical Research of Institut Guttmann.

The participants were anonymized and nonidentifiable. A specific written informed consent was not required for participants to be included in this study; nevertheless, at admission to Institut Guttmann, participants provided written informed consent to be included in research studies addressed by the Institut Guttmann hospital.

The Guttmann, NeuroPersonalTrainer web-based cognitive rehabilitation platform used in this study comprises a set of 149 different web-based cognitive rehabilitation tasks. There is no established previous order in which patients should execute such tasks. Therefore, every patient executed (eventually) a different subset of them in a different order during their rehabilitation process, taking between 2 and 6 months, distributed over two to five sessions a week. During each session, the patient executed between 4 to 10 cognitive rehabilitation tasks, and the total duration of one session ranged between 45 minutes to 1 hour. Each task mainly addressed one of the following functions: memory, executive functioning, attention, gnosias, calculus, orientation, language, and social cognition. Immediately after each execution of a task, the patient received a feedback on performance (ranging from 0-100, as the percentage of compliance), with 0% being the lowest level of compliance and 100% being the highest.

Before starting web-based cognitive rehabilitation using the Guttmann, NeuroPersonalTrainer platform, every patient was assessed once using standardized tests specifically validated for the population under study. Specific linguistic abilities were assessed using three subtests of the Barcelona test [32,33]: (1) repetition (maximum score=10), (2) denomination (maximum score=14), and (3) comprehension (maximum score=16). For assessing verbal fluency, the phonetic verbal fluency test [34] was used. The Trail Making Test was used to assess executive functioning [35] and the Wechsler Adult Intelligence Test–III [36] to assess visuospatial construction and perception.

In this study (without loss of generality), we assessed improvements in the memory cognitive function using the Rey Auditory Verbal Memory Test (RAVLT) [37] and the DIGITS subtest of Barcelona test [32]. RAVLT comprises three subtests: RAVLT learning (RAVLT075), free-recall memory (RAVLT015), and recognition (RAVLT015R). In RAVLT075, the patient was asked to recall as many words as possible from a list of 15 words, repeated five times. After a latency of 20 minutes, the patient was asked to recall the words (RAVLT015), and then the patient heard a list of 50 words containing the 15 initial sets that had to be recognized by the patient (RAVLT015R).

The DIGITS subtest (direct version) of the Barcelona test addresses working memory, and the patient was asked to repeat a series of numbers of variable lengths (3-9) until they failed in two consecutive series, reporting the largest series before failure [32].

The Elo rating system [19] is formally defined as [20]: given a rating estimate θ_i for each player i, the result of a match between players i and j is represented by R_ij∈{0,1}.

The actual ratings of each player are used to estimate the probability that player i wins:

which is used to update the ratings as follows, based on the Bradley-Terry model [38]:

where K is a constant parameter that controls how quickly θ_i changes, with large K values resulting in θ_i changing quickly and small K values resulting in θ_i changing slowly. In this study, we considered three extensions to the original Elo system: Glicko [39], Glicko-2 [40], and Stephenson [41]. Glicko models introduce a measure of reliability to assess the accuracy of the rating; that is, the rating deviation. Stephenson rating can be of interest in our context as it introduces a parameter that considers the strengths of the opponents, [41] being in our case, player i, the patient, and player j, the cognitive rehabilitation task.

Table 1 shows the demographic characteristics and clinical assessments of the 288 included patients.

The mean age at the time of the lesion was 51 (SD 9) years. The proportion of participants aged <65 years was 93.8% (270/288) (as opposed to most studies addressing ischemic stroke, we analyzed working-age participants). In relation to sex, in our data set, the proportion was 67.7% (195/288) men and 32.3% (93/288) women, which seems to suggest a bias in favor of men. Nevertheless, it somehow reflects reality in the general population, where the proportion of men experiencing ischemic stroke is larger than that of women [43-45]; however, women experience more hemorrhagic strokes [46].

We initially ran the four different Elo rating approaches (standard Elo, Glicko, Glicko-2, and Stephenson) in a reduced data set of 20 patients, each of whom executed the same task 20 times. Two screenshots of the selected task are presented in Figure 1. The task addresses executive functioning (planning), and the objective is to move a blue ball from an initial position in a maze to the final position, minimizing the number of moves. The bar at the right indicates the time left to perform the task. Figure 1 top shows the initial position of the ball, and Figure 1 bottom shows the status 20 seconds later when the objective was accomplished.

The included 20 patients were stratified into three categories according to their compliance in the maze task as follows:

Figure 2 presents the boxplots of the obtained results in the maze task at each execution in the 3 groups, showing their different levels of compliance.

We then ran the four Elo rating systems with default values for the initial ratings and K. We considered that when a patient gets a result >50%, they win the match against the maze; however, if their result is <50%, the maze wins. The ratings obtained using the Glicko approach are presented in Table 2. Patients are ordered in Table 2 according to their obtained ratings. Table 2 shows patients id19, id20, id16, id14, id17, id15, and id18 at the first seven positions. Similarly, patients from the midcompliance group are in positions 8-14, and patients from the low compliance group are in the bottom positions. The maze task itself is also considered as a player; it played all 400 matches, winning 118 and losing 282.

Figure 3 shows the obtained ratings using all four approaches for patient representatives of each of the compliance groups; we plotted id1 and id6 patients from the low-level group, id10 from the midlevel group, and id19 from the high level of compliance group to visualize how the Elo ratings represented their compliance levels.

We used the Stephenson rating with default parameters, considering the following criteria:

The Stephenson ratings were obtained by considering the first 50 task executions for every patient. We then stratified all 288 patients into 3 groups (each group comprised n=96 patients), according to their Elo ratings (low, middle, and high). Table 4 shows the memory assessments at admission and discharge, percentage of patients that improved, mean number of executed tasks, and obtained result comparisons for the three Elo levels (low, mid, and high) obtained using the 50 initial task executions for n=288 patients, with 96 patients in each Elo group that performed 38,177 task executions of the most frequent 68 tasks during rehabilitation.

Table 5 presents the obtained predictors of RAVLT075 at discharge (model 1), 54% of the variance explained and the obtained predictors of DIGITS at discharge (model 2), 43% of the variance explained.

When the Elo rating feature is excluded from model 1, it explains 52% of the variance, and when it is excluded from model 2, the resulting model explains 42%.

Table 6 presents the models used for predicting improvement in RAVLT075 and DIGITS. We used the criteria to decide whether a patient improved as described in the Dependent Variables section; 50.6% (146/288) of patients improved in RAVLT075, and 34% (98/288) of patients improved in DIGITS. We used the same Elo ratings as described in the Ranking Patients Using the Initial 50 Task Executions: Elo Rating section. Model 3 yielded an AUC of 0.73 (95% CI 0.64-0.82) for improvement in RAVLT075, with an accuracy=0.64 (95% CI 0.54-0.72), specificity=0.55, and sensitivity=0.73. Model 4 yielded an AUC of 0.81 (95% CI 0.72-0.89) for improvement in DIGITS, with an accuracy=0.73 (95% CI 0.64-0.81), specificity=0.22 and sensitivity=0.97. Models 3 and 4 are detailed in Table 6. When the Elo rating was excluded as an independent variable for model 3, the model yielded an AUC of 0.66 (95% CI 0.56-0.76) for improvement in RAVLT075, with an accuracy=0.62 (95% CI 0.52-0.71), specificity=0.62, and sensitivity=0.62. When the Elo rating was excluded as an independent variable for model 4, the model yielded an AUC of 0.73 (95% CI 0.62-0.83) for improvement in DIGITS, with an accuracy=0.72 (95% CI 0.62-0.80), specificity=0.34, and sensitivity=0.92. As shown in Table S3 (Multimedia Appendix 1), RAVLT075 was highly correlated with RAVLT015 and RAVLT015R at admission and at discharge.

To the best of our knowledge, in this study, Elo ratings were applied in the context of web-based cognitive rehabilitation tasks for the first time. We demonstrated the feasibility of using Elo ratings by using a publicly available R library (PlayerRatings) [41] on a synthetic use–case of 20 patients executing one task 20 times.

We then obtained the Elo ratings for each patient in a real rehabilitation setting where 288 adult patients with ischemic stroke followed cognitive rehabilitation, executing 68 different web-based rehabilitation tasks 38,177 times. We then performed a stratification of the patients into 3 groups (96 patients each) according to their Elo rating (low, middle, and high). We have shown the relationships among the three Elo rating levels and the proportion of tasks executed at three increasing difficulty levels (1, 2, and 3) with the rehabilitation outcomes in the memory cognitive function. We then developed four predictive models, where the Elo rating variables were significant in three of them (and quasi-significant in the fourth) for auditory verbal learning memory and working memory outcomes. We found that including Elo ratings as independent variables increased the model performance (for both linear and logistic regressions).

Several web-based cognitive rehabilitation platforms integrate some kind of stratification of patients as an initial step for treatment personalization. The web-based platform used in this study integrates an automatic therapy planning functionality—the intelligent therapy assistant (ITA) [47]. The ITA takes a set of patients’ cognitive profiles as the starting point, obtained using cluster analysis on the baseline cognitive evaluation. When a new patient starts cognitive training in Guttmann, NeuroPersonalTrainer, the ITA dynamically assigns the patient to the appropriate cluster. The ITA then schedules different cognitive tasks during a user-defined rehabilitation period for the new patient. Therefore, an important clinical implication of our results in this study involves the ITA (or any other data-driven therapy assistant) starting point: using Elo rating as a starting point, alternative to cluster analysis.

Obtaining an initial Elo rating for each patient is a simple process (in terms of both implementation and interpretation of results). As remarked in previous research, for example, in the field of educational tutoring systems, Elo rating use is encouraged because of its simplicity [20]. As shown in Table 3, the mean number of tasks executed by a patient in a session is 9, so in about five sessions (usually 2 weeks), an Elo rating for each patient obtained using the first 50 task executions will be available.

Therapists can then use the Elo rating to assign the patient to a skill level. In this study, in Table 4, we present the results using three skill levels, each of them with the same number of patients (96; or one-third of the N=288 total participants). Table 4 shows that, for example, 67% (64/96) of patients in the mid-Elo group improved in the RAVLT075 item, and 58% (56/96) of patients in the mid-Elo group improved in the RAVLT015 item. Meanwhile, for example, only 23% (22/96) of patients in the low Elo group improved in the DIGITS item. The low Elo group performed 29.1% (3,617/12,431) of their tasks at difficulty level 3, whereas the mid-Elo group performed 31% (3,859/12,449) of their tasks at difficulty level 2. This seems to suggest that patients in the low Elo group could have performed a higher proportion of tasks at difficulty level 1, which is more appropriate to their skills. Patients in the mid-Elo group performed a higher proportion of tasks according to their skill levels, which seems to be related to a higher proportion of patients obtaining improvements in the four memory items presented in Table 4.

Another clinical implication was noted on in a recent systematic review on computerized cognitive training [48]. The review highlighted the need to develop interventions focused on specific cognitive functions by means of concrete training or rehabilitation activities (or tasks). Our results contribute in that sense; considering, for example, model 1 for predicting RAVLT075 at discharge, we obtained a standard β=.09 for the Elo rating variable. Therefore, for every 113 points obtained in the Elo rating, an extra point in RAVLT075 at discharge is obtained. If we consider, for example, in the maze task presented in Figure 1, patient id12 (Elo ranking=2240) and patient id8 (Elo ranking=2112) presented in Table 2, the difference between their Elo ratings is 128 points, with both patients belonging to the intermediate compliance group. Similar Elo rating scores were obtained for the final sample of N=288. Therefore, therapists can identify at the early stages of the rehab process–specific cognitive tasks where patients are close to obtaining a draw or a win (result ≥40%) and address different strategies [48] to improve performance in such specific tasks.

Several limitations to the study need to be highlighted. First, we conducted a single-center study, an advantage of which is that data were obtained and included by clinicians trained in neurological rehabilitation, and all patients were managed under the same stroke rehabilitation protocols. The Guttmann, NeuroPersonalTrainer platform has already been integrated into the clinical practice of several acquired brain injury centers; nevertheless, their patients were not included in this analysis. A multicenter stroke study may include an initial preprocessing phase, wherein patients are grouped according to their initial National Institutes of Health Stroke Scale severity to avoid additional heterogeneity. Thereafter, Elo rating techniques, such as those proposed in this study, maybe applied within such groups. External validation assessments common to all participating centers are also an important aspect to be addressed in this future multicenter study. Second, the studied health area belongs mainly to the urban population, with a small rural population or populations from other regions. Third, our analysis lacked computerized tomography or magnetic resonance imaging examinations that describe the presence of contusion, hematoma, hemorrhage, ischemia, or other signs of parenchymal lesions in the frontal, temporal, parietal, occipital, and cerebellar lobes or diffuse axonal injury.

Fourth, our sample did not include any patients with missing data. All data used as inputs were complete. Fifth, our analysis did not include indicators of mental health or other comorbidities. Persons who experience a stroke may have one or more preexisting medical comorbidities at the time of injury (eg, alcohol use and depression). Therefore, we plan to include comorbidity analyses in future research studies. Sixth, in all our Elo rating calculations, we used the default value for the K constant. Several approaches to K optimization have been reported, such as hill climbing, gradient descendent, or Bayesian [20], which can also be addressed in future work. Finally, the criteria for defining wins, draws, and losses in our Elo ratings were also constant for every task, and another possible improvement could be to fit such criteria according to the task difficulty level, considering the strength of the opponents (patients’ skills and task difficulty levels) that can be addressed using the Stephenson extension [41].

Cluster analysis has been extensively proposed in previous research to address heterogeneity in patients with acquired brain injury [49-51] and as an initial step for patient profiling. Most previous studies use commercial software products for cluster analysis, which are, in turn, not integrated into the web-based cognitive rehabilitation platform.

In a recent study, Faria et al [52] presented a framework for the creation of personalized cognitive rehabilitation tasks based on a participatory design strategy. They selected 11 paper-and-pencil tasks from standard clinical practice and parameterized them with multiple parameter configurations. A modeling approach was used to quantitatively determine how the task parameters affect each of the cognitive domains (memory, executive functions, attention, and language). For modeling this relationship, the parameters of each task were used as predictors of the demands in each cognitive domain. In our case, the parameters of each task were used by experts to assign a difficulty level to each task (difficulty level 1, 2, and 3, as presented in Table 4), where each task aims to address one main cognitive domain (memory, executive functions, attention, and language).

We have shown the feasibility of Elo ratings for identifying patients’ profiles at the early stages of cognitive rehabilitation in a real clinical setting. Elo ratings can be used to match skills with task difficulties, aiming to maximize improvements in specific cognitive functions. Such Elo ratings are also significant in predicting cognitive outcomes. Elo ratings increased the models’ performance (for both linear and logistic regressions). Generalization of the use of Elo ratings beyond patients with stroke to any other population with acquired brain injury requiring cognitive rehabilitation in any web-based platform is straightforward because of the simplicity of existing open-access Elo rating implementations.