This secondary analysis utilized data from 2 prospective studies evaluating clinical and engagement-related outcomes in patients with musculoskeletal (MSK) conditions. Data were collected from June 2020 to July 2023. This prognostic study adhered to the TRIPOD-AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis-AI) reporting guidelines (Table S1 in Multimedia Appendix 1).

All research was conducted in accordance with the relevant guidelines and regulations set forth by the Declaration of Helsinki. This study received prospective approval from the New England institutional review board (120190313) and Advarra institutional review board (Pro00063337) and was registered on ClinicalTrials.gov (NCT04092946 and NCT05417685) on September 17, 2019, and June 14, 2022, respectively.

All participants provided electronic informed consent to participate in the study, with a waiver of documentation of consent approved by the New England and Advarra institutional review boards.

All collected data underwent a rigorous anonymization process to safeguard the privacy of the individuals involved in the research. The data collection and analysis methods complied with relevant guidelines and regulations. Participants were not offered any form of compensation.

Beneficiaries of employer health plans from all US states applying to Sword Health’s DCPs were included. The inclusion criteria were as follows: acute or chronic LBP, an average Numerical Pain Rating Scale (NPRS) score of ≥4/10, and at least one NPRS reassessment during the intervention. The exclusion criteria were as follows: (1) health conditions (eg, cardiac or respiratory) that are incompatible with at least 20 minutes of light-to-moderate exercise; (2) cancer-related back pain or current cancer treatment for a non-MSK condition; and (3) serious neurological signs or symptoms, including bowel or bladder dysfunction. All participants provided informed consent.

The DCP consisted of exercise, education, and cognitive behavioral therapy for up to 12 weeks, with a default recommendation of 3 sessions per week, as described elsewhere [22], after enrolling through a dedicated website. Patients completed a baseline condition form that included information on their demographic and clinical characteristics and selected their PT based on their preferences. Subsequently, an onboarding video call was conducted in which the PTs gathered additional medical history and established goals through shared decision-making. Each patient received a Food and Drug Administration–listed class II medical device, which included a mobile app on a dedicated tablet (containing the program) combined with motion tracking, as well as a cloud-based portal that enabled asynchronous remote monitoring and treatment prescription by the PT. A tailored educational program and cognitive behavioral therapy were also provided [25-27]. Bidirectional communication with the PT was facilitated through a secure chat feature on the smartphone app and video calls.

Considering the overarching intervention goal of promoting improvements in pain levels in daily living, pain response was selected as the primary outcome. Pain was assessed using the 11-point NPRS with a 7-day recall period during sessions 9, 18, or 27.

Pain response, defined as at least a 30% reduction in the NPRS by the end of the program, aligns with clinical recommendations from the Initiative on Methods, Measurement, and Pain Assessment in Clinical Trials (IMMPACT) [28]. This definition is supported by a consensus meeting that interpreted the clinical significance of treatment outcomes in clinical trials for chronic pain treatments. Additionally, we included the criterion of concluding the program with an NPRS score equal to or below 3 points (the threshold for mild pain) [29] to account for patients who started with lower pain levels and ended the program with acceptable pain levels. The widespread use of this binary outcome, along with its practicality in real-world settings, supported its inclusion.

The variables used for model development were selected based on existing literature and their clinical relevance to MSK pain. A detailed description of the variable domains and their acceptable ranges can be found in Table 1. The data set variables included (1) demographic characteristics (eg, race and ethnicity, social deprivation index) [30]; (2) baseline clinical presentations (eg, acuity, disability, mental health, productivity, fear-avoidance beliefs, presence of red flags); (3) time series of mean active range of motion (ROM) throughout all repetitions of each exercise during each session; (4) session-related usability (eg, movement errors, number of exercises, sets and repetitions, the time between sessions, session duration); and (5) self-reported fatigue and pain experienced during exercises, measured on a 0-10 scale (Figure 1).

All data were automatically stored in the cloud portal. Demographic and clinical characteristics were collected through an onboarding form using validated PROMs. ROM and session usability were passively collected, while fatigue and pain experienced during exercise sessions were recorded at the end of each session.

Given that there is no established method to calculate sample sizes for prognostic models using ML, we followed the recommended guideline for standard model development, which suggests obtaining 10-20 events per predictor parameter. As we estimated including up to 300 predictor variables in the models, a sample size range of 3000 to 6000 was calculated.

The overall missingness was 18.45% (324,385/1,757,875; see Table S2 in Multimedia Appendix 1), comprising the following: 17.07% (300,117/1,757,875) missing completely at random, where the missing data points were attributed to system settings; 0.29% (5169/1,757,875) missing at random, where the missingness could be explained by other variables (eg, specific protocol prescriptions that did not include one of the studied exercises); and 1.09% (19,099/1,757,875) missing not at random, where the reasons for the missing data were related to unobserved factors. Categorical variable imputation was performed by creating a new category labeled “not available,” followed by dummy encoding. Continuous variables were imputed using multiple imputations by chained equations or by assigning a value outside the distribution (eg, –1), depending on the nature of the missingness. Multiple imputations can provide unbiased estimations for data that are missing completely at random and at random. As the rate of missingness for not-at-random data is very small, we used multiple imputation techniques to handle the missing data overall. The cleaned data set included 6125 patients and contained a number of variables ranging from 235 at session 1 to 275 at session 7.

Demographic and clinical characteristics identified as statistically different between the training and test sets were subjected to subgroup analysis to assess potential bias, including factors such as gender, social deprivation index, age, acuity, and pain levels.

The model output classified patients as responders or nonresponders, based on the previously defined outcome, and was contingent on the applied threshold (eg, 0.95).

A data split was conducted in which 70% (4313/6125) of the patients were randomly assigned for training the model, while the remaining 30% (1812/6125) were reserved for testing.

Demographic and clinical presentations at baseline were analyzed using means and proportions. The minimum and maximum ROC-AUC values were calculated by bootstrapping the training and test sets separately (1000 bootstrapped sets). Model comparisons involved ROC-AUCs calculated using the DeLong algorithm [43]. Feature engineering models were developed using R (version 4.2.2; R Foundation for Statistical Computing; packages: lavaan, mlVAR, xgboost, mice, and tidySEM). All other analyses were conducted using Python (version 3.9.7; package: TreeSHAP). The level of significance was set at P<.05, considering 2-sided hypothesis tests.

A total of 6125 patients met the inclusion criteria (see the flowchart of the study cohort in Figure S3 in Multimedia Appendix 1). Among them, 2172 (35.46%) were classified as nonresponders, while 3953 (64.54%) were responders. The analysis encompassed various domains and subdomains, including demographic, socioeconomic, productivity, clinical, prescription, exercise performance, and motivation/compliance data, up to session 7 (see further details in Table 1). Baseline demographic and clinical characteristics for the overall cohort, as well as stratified by training and test data sets, are presented in Table 2. The overall cohort primarily consisted of women (3445/6125, 56.24%), patients aged 41-60 years (3488/6125, 56.95%), fully employed individuals (5119/6125, 83.58%), and college graduates (4037/6125, 65.91%). Additionally, 2335 (38.12%) individuals were classified as obese, 1931 (31.53%) identified as racial or ethnic minorities (ie, Asian, Black, Hispanic, and other), and 1585 (25.88%) came from socially deprived backgrounds.

Overall, the cohort reported moderate pain, with an NPRS score of 5.7 out of 10.0 (SD 1.4), and a disability score on the Oswestry Disability Index (ODI) [44] of 23.1 (SD 12.7) at baseline. This was primarily associated with chronic pain, defined as pain lasting 3 months or longer (4957/6125, 80.93%). Severe pain (NPRS: 8-10) was predominantly reported by patients with acute LBP (168/1168, 14.38% vs 519/4957, 10.47%, in chronic cases, P<.001). High fear-avoidance beliefs (Fear-Avoidance Beliefs Questionnaire score ≥15 [45]) were reported by 1391 of the 6125 (22.71%) patients of the cohort, while 2144 (35.0%) and 1590 (25.96%) patients reported at least mild anxiety (7-item General Anxiety Disorder Scale [GAD-7] score ≥5 [31]) or depression (9-item Patient Health Questionnaire [PHQ-9] score ≥5 [32]), respectively. A large majority (4900/6125, 80.0%) reported impairment in daily activities (Work Productivity and Activity Impairment Questionnaire for General Health [WPAI]—activity score >0), while 3296 of the 6125 (53.81%) patients reported either presenteeism or absenteeism (WPAI overall score >0). Clinical outcomes were similar across both the training and test data sets, with the exception of the ODI, which was statistically higher in the test data set (22.6, SD 12.5 vs 24.4, SD 13.2), although this difference was not clinically meaningful [46].

The LGCA revealed a significantly different latent representation of trunk motion over time for responders and nonresponders (P<.001; Figure S4 in Multimedia Appendix 1). Similarly, the temporal structural network model showed 2 significantly different temporal correlation patterns of trunk motion for responders and nonresponders (P<.001; Figure S5 in Multimedia Appendix 1).

The best models from both methodologies (LightGBM and RNN), following hyperparameter optimization, resulted in similar ROC-AUC scores in the test set across all time points (P=.98, .52, .69, .81, .38, .68, and .22, respectively, for sessions 1-7; Figure 3A and B). Model performance improved over time for both models (P<.001 for both, DeLong algorithm), with predictions at session 7 reaching an ROC-AUC of 0.70 (95% CI 0.65-0.71, based on bootstrap resampling; Table S3 in Multimedia Appendix 1) for RNN and 0.71 (95% CI 0.67-0.72) for LightGBM, compared with 0.66 (95% CI 0.61-0.68) and 0.67 (95% CI 0.61-0.67) obtained in session 1. These results corresponded to a precision-recall AUC at session 7 of 0.56 (LightGBM: 95% CI 0.49-0.58; RNN 95% CI 0.48-0.57; Figure 3C and D) and a weighted F₁-score of 0.68 (95% CI 0.64-0.69) for both models.

For the models described, considering a precision of 70%, the LightGBM achieved a specificity of 0.97 (95% CI 0.95-0.98), a sensitivity of 0.13 (95% CI 0.02-0.25), and a negative predictive value of 0.68 (95% CI 0.68-0.68). Similarly, the RNN achieved a specificity of 0.97 (95% CI 0.94-0.97), a sensitivity of 0.12 (95% CI 0.00-0.24), and a negative predictive value of 0.68 (95% CI 0.65-0.71).

Considering the best models, we analyzed the top 20 outcome predictors at each session based on absolute mean SHAP values [41]. Figure 4 depicts cumulative values of features aggregated by domains over time for the LightGBM (Figure 4A) and RNN (Figure 4B). Detailed feature stratification for both models is available in Figures S6 and S7 in Multimedia Appendix 1, respectively.

Although both models incorporated features from comparable domains, the importance of each feature varied depending on the model. The flagged domains included pain-associated metrics, exercise performance, and motivation and compliance data, which utilized real-time passive data collection.

Although the pain domain was highly predictive for both models (Figures 4A and B), LightGBM placed greater emphasis on exercise performance, motivation, and compliance data than the RNN. Within the exercise performance domain, the most informative features included time spent exercising, exercise accuracy, and the execution of specific exercises related to trunk motion. Notably, lower training time or poorer performance in particular sessions was associated with an increased likelihood of nonresponse (Figure S8A in Multimedia Appendix 1). Low motivation and compliance, specifically characterized by low exercise consistency—evaluated by longer intervals between sessions (>3 days, Figure 4D)—was associated with a higher likelihood of being a nonresponder. This observation was consistent across other variables reflecting motivation toward compliance, such as the time between registration and program start, as well as the reasons preventing patients from completing the program.

The RNN was more reliant on the temporal patterns of mental health and other clinical variables (eg, medication intake, intent to undergo surgery) for its predictions. The presence of mental distress was associated with a higher likelihood of being a nonresponder across all sessions, as exemplified by the item “Feeling nervous, anxious, or on edge” from the GAD-7 scale (Figure 4E). Even though LightGBM did not rank mental health features among the top 20 predictors, higher GAD-7 and PHQ-9 scores were correlated with poorer outcomes, particularly in the later sessions (eg, Figure S8B in Multimedia Appendix 1). Additionally, other clinical features, such as prescribed medication, were associated with a lower probability of being a responder.

Both models utilized features related to individual characteristics, including demographics and socioeconomic status. Subgroup analyses evaluating potential imbalances between the training and test sets indicated that similar results could be obtained regardless of demographic or clinical characteristics (Table S4 in Multimedia Appendix 1). Overall, women and individuals from more socially deprived areas, particularly those from regions with higher unemployment rates, had a worse prognosis compared with their counterparts (Figure S8C and S8D in Multimedia Appendix 1).

Figure 5 illustrates an example of the output from the tool provided on the PT portal for 2 patients during session 4. This output includes the model’s prediction of nonresponders (ie, true positives, with an RNN precision of 70% on the test set) and the key drivers behind this classification. For patient A, the classification is primarily influenced by high pain reported during exercise in session 4 (NPRS = 6 out of 10), baseline characteristics indicating severe clinical presentation (such as high ODI and mental distress), and self-reported potential barriers to motivation. Patient B also reported high pain during exercise in session 4 (NPRS = 8 out of 10) and exhibited low exercise performance for that session (3.5 out of 5 stars). Additionally, patient B indicated prescribed medication intake (eg, opioids) and the presence of at least one clinical red flag, despite being cleared by a physician to participate in the program. With an ODI score of 50 and low educational attainment, this information is subsequently analyzed by the PT, allowing for necessary refinements to the intervention. This may involve adjusting exercise intensity or type or prioritizing specific components of the DCP, such as education and cognitive behavioral therapy. These adjustments will subsequently impact future model predictions in a dynamic closed-loop optimization that contributes to the refinement of the tool.

This study leveraged passively and automatically collected patient data from a digital MSK intervention to develop a predictive tool that assists PTs in optimizing treatment. These models utilize continuously updated data from each enrolled patient to identify early those who are less likely to achieve a clinically meaningful reduction in pain. Model performance significantly increased over time, achieving an ROC-AUC of 0.70 (95% CI 0.65-0.71, from bootstrap resampling) for the RNN and 0.71 (95% CI 0.67-0.72) for the LightGBM at session 7. These models identified features from domains related to pain, exercise performance, motivation, and compliance as the most predictive of outcomes. This information can inform PTs for continuous treatment adjustments to maximize response rates. This predictive tool operates through a dynamic loop, and when applied in clinical practice, it continuously enhances treatment refinement and improves tool performance over time.

Previous ML models designed to support LBP management without real-time data collection have reported performances (ROC-AUCs) ranging from 0.49 to 0.71 [6,7,9,10,21], with 1 model achieving a higher AUC of 0.84, but only near the end of treatment [11]. Additionally, the currently used STarT Back prediction tool has reported AUCs of 0.63 and 0.60-0.62 when predicting pain intensity at 12 weeks and 1 year, respectively [6,7]. The wide range of reported model performances reflects not only the challenges associated with each proposed objective but also the level of uncertainty that these tools generate. In this study, both models’ performances significantly improved over time, achieving an AUC of 0.70-0.71 at session 7 using real-world data. This AUC suggests that pain response at an early stage of the intervention is partially explained by the variables used. However, as treatment progresses, it is plausible that model performance will continue to improve toward the end. As an example, Brennan et al [11] demonstrated a 1.3 times increase in model performance when using PROMs collected at session 3 to predict outcomes at session 6. Our study aimed to be less reliant on PROMs and more focused on a scalable solution capable of improving over time [47,48]. While specificity and sensitivity depend on the threshold applied, the results presented here align with those previously reported [6,7,9-11,21]. The choice of threshold to balance precision and recall must align with the clinical setting’s needs, taking into account the costs and implications of false positives and false negatives.

In addition to real-time predictions, these models may offer timely insights that enable PTs to determine the need for referring patients to specialized care (eg, escalation to psychologists) through shared decision-making with the patient. Both models identified similar feature domains, including pain-associated metrics, exercise performance, motivation, and compliance, thereby surpassing the predictive value of individual demographic prognostic factors. These domains cannot be captured solely through PROMs, further emphasizing the relevance of such models in managing LBP.

To predict the pain that patients experience during daily activities, the model incorporates pain levels reported during exercise, alongside baseline clinical presentations (including the NPRS, the “pain intensity” item from the ODI, and acuity)—features previously established as poor prognostic indicators [49-51]. This underscores the importance of pain experienced during exercise as a predictor of outcomes. Specifically, exercising with pain levels exceeding 4 out of 10 has been shown to predict negative outcomes [52], suggesting that these patients may benefit from adjustments to their exercise prescriptions. The use of analgesic medications, such as opioids, has also been linked to a poorer prognosis, as noted in previous studies [53-55].

Aside from pain, both models prioritized motivation, compliance, and exercise performance data for predictions. Regarding motivation, indicators such as self-reported resistance to commit to the intervention, delayed program initiation, and extended intervals between sessions (>3 days) were associated with a lower likelihood of response, reinforcing previous findings [56]. In the exercise performance domain, the models emphasized exercise execution data, including the time dedicated to sessions, performance scores, and the number of incorrect exercises, with lower scores linked to worse outcomes. Challenges associated with compliance frequently contribute to increased frustration and decreased adherence to treatment [57,58] and have been previously described as risk factors for poor outcomes [5,59]. Co-occurring with low adherence rates, anxiety and depression were associated with a lower likelihood of response in one of the models (RNN). This finding is unsurprising, considering the well-established negative feedback loop between mental health and MSK pain [60,61].

Although the demographic and socioeconomic profiles of patients are relevant, they may be of lower importance than the previously mentioned domains. Certain groups, such as women and the unemployed, may encounter systemic barriers that hinder their ability to achieve pain relief.

In summary, implementing this tool would enhance care coordination and patient management by identifying individuals at higher risk for poor pain outcomes. This, in turn, can streamline workflows, fostering informed decision-making and tailored treatment programs. The continuous stream of data generated by patients enables refined predictions at each interaction, allowing PTs to make timely adjustments to the program. These adjustments could then be fed back into the system to further enhance its recommendations. To assess potential biases, the subgroup analyses conducted in this study focused on demographic characteristics and socioeconomic factors, indicating a low risk of bias. However, further evaluation of algorithmic bias will be necessary once the model is fully operational and deployed in real-world settings. Continuous monitoring will be essential to ensure that the model does not disproportionately impact specific populations or reinforce existing health disparities.

While there may be concerns among PTs regarding the “black box” nature of AI, model explainability will be crucial in overcoming this barrier. Before full implementation, training for PTs (or clinicians) is essential to ensure they understand and can effectively use the tool, thereby mitigating fears and preventing erroneous use of AI. Nevertheless, the design of this AI tool ensures that final clinical decisions remain with the PT, preserving human judgment and critical reasoning. This approach can foster trust in the system and promote the safe integration of AI into clinical workflows.

Future research on the development of this predictive tool should concentrate on clinically validating the closed feedback loop. Additionally, optimizing the performance of the AI tool may be further enhanced by incorporating supplementary data sources, such as real-life context data variables obtainable from smartwatches (eg, pedometry and sleep quality) and clinical context information (eg, presence of specific comorbidities and clinical records). This could reveal additional data patterns and, consequently, contribute to improved model accuracy. With the nearly exponential increase in data, it is plausible that RNNs or other deep sequence models will continue to enhance their performance and eventually surpass LightGBM, given their natural ability to handle sequential data [47,48]. Additionally, the integration of continuous volumes of data may expand the model’s predictive capabilities to assess long-term outcomes. In particular, for LBP, predicting long-term outcomes will help determine which types of interventions lead to more sustained recovery with functional improvement, and reduce the likelihood of relapse or chronicity over time.

Finally, as the AI model relies on protected health information, implementing robust safeguards and establishing data stewardship protocols are essential to maintain confidentiality, integrity, and security.

Other future considerations include exploring the application of such tools to other MSK conditions.

This study has important strengths. First is the novelty of exploring how data collected from a DCP can be applied in an ML model to continuously predict patient outcomes and foster personalized care. Second, this cohort’s demographic and clinical characteristics mirror that of the US population reporting LBP [62,63], with the percentage of nonresponders for pain being consistent with previous studies evaluating in-person and digital LBP management [11,22,64,65]. Finally, the nature of the data that are automatically and passively collected, which are less reliant on PROMs, makes it easy to obtain and less burdensome for both patients and clinicians. However, this study also has limitations that warrant noting: (1) Although the cohort was large and decentralized, we cannot dismiss the possibility of overfitting; additionally, as the population focuses on beneficiaries of health plans, this may limit generalizability to populations not covered by health insurance. Therefore, extended testing on new cohorts is critical for the external validation of the AI tool. Such validation would provide a more accurate assessment of the model’s performance, ensuring that its effectiveness and robustness are maintained across different populations that may enroll in the program. Additionally, it could help identify areas in need of improvement. Furthermore, external validation studies should be followed by real-world applications to evaluate the usefulness of the deployed model in clinical settings; (2) despite being highly used and recommended by clinical guidelines, transforming the pain outcome into a binary classification may obscure subtle variations in patient responses. This simplification could lead to misclassification of patients who are close to the threshold. Future approaches could consider modeling the outcome as a continuous variable. Additionally, developing a composite score that reflects improvements across multiple core outcome domains—including Patient Global Impression of Change and Activities of daily living functionality—could better capture the multifactorial nature of LBP and provide a more comprehensive assessment of patient improvement in a single metric; (3) we used an extensive number of features that capture critical domains in the rehabilitation realm; however, other potential features, such as measures of self-efficacy and indicators depicting transitional states between clusters of trajectories, which could hold high predictive power, were not included. Incorporating these features in future designs could enhance the model’s predictive accuracy; (4) finally, the early stage of this tool’s development hampers the ability to assess its impact on clinical outcomes. Future steps should focus on implementing and evaluating the tool’s dynamic loop to better understand its effectiveness and feasibility, as well as to ascertain its net benefits.

This study underscores the potential of a predictive tool leveraging ML models to enhance the management of LBP in a digital setting, redirecting care pathways early on and throughout the treatment course, thereby fostering personalized care. The application of such a strategy is particularly important for managing a heterogeneous population with diverse phenotypes across different cultural contexts.