Knee osteoarthritis (KOA) is a chronic musculoskeletal disease with a highly heterogeneous clinical course; even among patients with similar baseline symptoms or radiographic severity, subsequent changes in pain, function, and structure may follow different trajectories [1,2]. The Western Ontario and McMaster Universities Osteoarthritis Index (WOMAC) and the Kellgren-Lawrence (KL) classification are commonly used measures in clinical assessments and longitudinal studies. Accurately estimating future changes in these measures aids in risk stratification, follow-up planning, and individualized management. In response to this need, various disease-specific models developed in recent years have demonstrated the potential to predict pain and structural progression in KOA [3-6].

Large language models (LLMs) can extract clinical information from unstructured data—such as symptom descriptions, medical histories, treatment histories, and imaging reports—and perform tasks such as medical question answering, clinical reasoning, and information summarization [7,8]. In the field of osteoarthritis, studies have shown that LLMs can identify hip osteoarthritis or KOA from questionnaires, generate diagnostic and treatment recommendations generally consistent with those of specialists using anonymous medical records, and convert free-text knee magnetic resonance imaging reports into structured scores [9-11]. Studies indicate that LLMs can extract information relevant to a patient’s current clinical condition from clinical narratives and convert it into clinically meaningful, structured judgments; however, it remains unclear whether they can be further used to predict individual outcomes.

Although models such as DeepSeek have demonstrated strong clinical reasoning capabilities in diagnosis and treatment decision-making for multidisciplinary cases [12], longitudinal outcome prediction differs in nature from tasks involving the identification, induction, or classification of existing information. Therefore, the evaluation of LLMs’ longitudinal predictive capabilities must be based on outcomes observed in real-world follow-up data. Previous research on medical LLMs has emphasized the need to expand the scope of evaluation beyond simple accuracy to include real-world clinical data, workflow adaptability, and multidimensional performance metrics [13,14]. Furthermore, generative models may produce different outputs for the same input [15]. It is still necessary to validate whether LLMs can reliably predict longitudinal outcomes for KOA based on clinical narratives while maintaining stable outputs in a real-world follow-up cohort.

This study aimed to evaluate DeepSeek’s ability to identify KOA and assess baseline WOMAC scores and report-derived KL grades from standardized clinical interview records and descriptive radiology reports. We also aimed to compare its predictions of 6-month WOMAC scores and report-derived KL grades with those of one rheumatologist and a persistence model and assess the reproducibility of its outputs across 3 independent runs.

This was a single-center prospective observational cohort study. A rheumatologist predicted the 6-month outcomes at baseline. The LLM evaluation was conducted retrospectively after completion of follow-up. Patients were consecutively enrolled from January to November 2025 in the Rheumatology and Immunology Department and the Orthopaedics Department at Fuxing Hospital, which is affiliated with Capital Medical University. Patients were eligible for enrollment if they had a definitive diagnosis of primary KOA according to the Guidelines for the Diagnosis and Treatment of Osteoarthritis (2018 edition) [16], completed a standardized baseline interview, and underwent anteroposterior and lateral radiographs of both knees. Patients who could not be definitively diagnosed, those with secondary KOA, or those with insufficient baseline data to complete the physician and LLM assessments were excluded. The primary analysis was limited to patients who completed follow-up and had complete outcome data. The overall study design and evaluation process are shown in Figure 1A. Reporting of the LLM evaluation was guided by the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Large Language Model version) checklist [17].

Researchers trained according to a standardized protocol conducted baseline assessments using a standardized interview guide and recorded patients’ responses in natural language. The interview guide and recording template are provided in Multimedia Appendix 1. Prior treatment was defined as receiving at least one KOA-related intervention within the 3 months prior to baseline, including medication, intra-articular injections, physical or rehabilitation therapy, and the use of assistive devices. Regular exercise was defined as exercising 4 or more times per week for 30 or more minutes per session; frequent stair climbing was defined as climbing 4 or more times per day, with each session involving 30 or more steps or 2 flights of stairs; and weight control was defined as actively undertaking dietary or exercise interventions to manage body weight.

The 96-point WOMAC scale was used to assess knee pain, stiffness, and physical function [18,19]. At the time of submission to the physician and the LLM for evaluation, all interview data were retained as natural language text without numerical coding. Responses containing explicit severity ratings were scored according to the WOMAC scale’s categories from 0 to 4 points. When descriptions were ambiguous, the physician and the LLM scored them independently based on semantic analysis. The same procedure was followed for the 6-month follow-up, which was conducted via outpatient visits, hospital admissions, or telephone calls, with the follow-up window set at 6 months ±15 days after baseline.

Baseline and 6-month follow-up examinations followed a standardized protocol, with anteroposterior and lateral radiographs of both knees acquired using the same digital radiography system (Discovery XR656; GE Healthcare). The imaging data were independently reviewed by 2 radiologists with 5 and 8 years of experience, who each prepared descriptive reports. In cases of disagreement, a third radiologist with 15 years of experience reviewed the images and made the final determination. The radiologists were blinded to the time of examination, physician predictions, LLM outputs, and clinical outcomes; they described only radiographic findings and did not directly assess the KL grade. The KL grades at baseline and 6 months were determined by the respective clinical evaluators based on interview data and radiology reports.

The baseline assessment, based on comprehensive physician interview records and imaging reports, was used to confirm the diagnosis of KOA. For patients with bilateral KOA, the target knee was selected according to predefined rules: priority was given to the side with a higher KL grade; if the grades were the same, the side with more severe symptoms was selected; if they were still the same, the right knee was selected. Once the target knee was determined, it remained unchanged, and all baseline and follow-up outcomes were assessed based on that side.

A board-certified rheumatologist with 10 years of clinical experience (physician A) independently reviewed standardized baseline interview records and imaging reports with diagnostic conclusions redacted to establish the diagnosis of KOA; assess baseline WOMAC subscores, total scores, and KL grading; and predict corresponding 6-month outcomes. All assessments and predictions were based solely on baseline data; physician A remained blinded to the actual 6-month outcomes and LLM outputs. The 6-month outcomes were independently assessed by another rheumatologist (physician B) with 25 years of clinical experience based on follow-up interview records and imaging reports from the same period, including the WOMAC subscores, total score, and KL grading. Physician B was blinded to physician A’s assessments and predictions, as well as the LLM outputs.

On June 19 and 20, 2026, DeepSeek was accessed via the official web interface in Instant Mode (DeepSeek-V4-Flash) [20]. Deep Thinking and Smart Search were left enabled by default. No search activity was visible. The exact back-end build and sampling parameters were unverified. The model was not fine-tuned. Both the LLM and physician A used exactly the same deidentified baseline data, including standardized interview records and imaging reports with diagnostic conclusions removed; neither had access to the original x-ray images. A standardized prompt was used, instructing the model to determine the most likely diagnosis; assess the baseline WOMAC subscales, total score, and KL classification; and predict the corresponding 6-month outcome. The affected side was extracted from the diagnostic responses provided by both physician A and the LLM; cases where the diagnosis was correct but the side was not specified were classified as “side inconsistency.” The prompt specified the use of the WOMAC scale from 0 to 96 points and required the output to be a single, unambiguous numerical value or grade. The standardized interview template and complete 4-step prompt sequence are provided in Multimedia Appendix 1.

To evaluate interrun reproducibility, 3 independent runs were conducted for 111 patients. For each run, a new session was created using the same input and prompts, with consistent case order and no prior run information provided. Reruns were required only for out-of-range WOMAC scores or an incorrect scale version; none were needed. All 333 LLM outputs were structured, extracted, and entered by an independent evaluator and then cross-checked against the original responses one by one by a second evaluator.

Radiographic progression was defined as an increase of 1 or more grades in report-derived KL at 6 months relative to the baseline grade assessed by physician A. The direction of change in the WOMAC score was classified as a decrease (<0), no change (0), or an increase (>0) based on the difference between the 6-month total score and the baseline total score. In the persistence model, the baseline WOMAC score and KL classification assessed by physician A were used as the corresponding predictors for the 6-month values.

Continuous variables are expressed as means and SDs or medians and IQRs, and categorical variables are expressed as frequencies and percentages. Paired comparisons between baseline and 6-month data were performed using the Wilcoxon signed-rank test. Correlation between continuous outputs was assessed using the Pearson correlation coefficient. Agreement was evaluated using the intraclass correlation coefficient (ICC(2,1); 2-way random effects, absolute agreement, and single measurement) and Bland-Altman plots. Ordered categorical outputs were evaluated using exact agreement rates and the linear-weighted κ coefficient. Predictive performance was characterized using the mean absolute error (MAE), root mean squared error (RMSE), and signed error, and calibration was assessed using the calibration slope and calibration curves. The 95% CI for the MAE was estimated using 5000 bias-corrected and accelerated bootstrap repetitions.

The 3 LLM run results and the prediction errors of different methods were compared using the Friedman test; when overall differences were significant, the Wilcoxon signed-rank test was performed using the Holm correction method, and the performance of imaging progression identification was evaluated using sensitivity, specificity, positive predictive value, negative predictive value, and balanced accuracy. Overall and pairwise comparisons between methods were performed using the Cochran Q test and the exact McNemar test, respectively. A linear mixed-effects model with patient-specific random intercepts was used to analyze factors associated with the absolute prediction error rates of the LLM. All tests were 2 sided; P<.05 was considered statistically significant. Statistical analysis and graphing were performed using SPSS Statistics (version 27.0; IBM Corp) and Prism (version 11; GraphPad Software), respectively.

Missing data were assessed using the Little missing completely at random (MCAR) test and comparisons of baseline characteristics by follow-up status. The primary analysis included 111 completers; inverse probability weighting (IPW) was used for sensitivity analysis. Multiple imputation was considered but not performed.

This study was approved by the ethics committee of Fuxing Hospital, Capital Medical University (approval 2025FXHEC-KSP088). Written informed consent was obtained from all participants. Deidentified baseline interview records and radiology reports were used for the physician and LLM evaluations. Participants received no compensation for taking part in the study.

This study screened a total of 186 patients; 25 refused x-ray examination, 11 could not be definitively diagnosed with KOA, and 22 had secondary KOA. A total of 128 patients met the inclusion criteria, of whom 17 were lost to follow-up; ultimately, 111 patients were included in the analysis (Figure 1B). The mean age of the analyzed patients was 56.8 (SD 11.1) years; 60.4% (67/111) were female, and 63.1% (70/111) had bilateral knee involvement (Table 1). The baseline WOMAC total score was 19 (IQR 13-25) points, and at 6 months, it was 21 (IQR 13.5-27) points; the difference was not statistically significant (P=.16; Table 2). At both baseline and 6 months, KL grade 2 was the most common, accounting for 69.4% (77/111) and 66.7% (74/111), respectively. During follow-up, 13.5% (15/111) of the patients experienced radiographic progression, 85.6% (95/111) maintained stable grading, and 0.9% (1/111) experienced a decline in grading.

Both 6-month outcomes were missing for 13.3% (17/128) of the participants, who could not be contacted or did not return. Participants lost to follow-up had lower baseline WOMAC scores and less regular exercise (P=.009 and P=.002, respectively; Table S1 in Multimedia Appendix 2). The Little test yielded χ²₃=7.1 (P=.07). The main WOMAC performance ordering was unchanged in the IPW sensitivity analysis (Table S2 in Multimedia Appendix 2).

Across the 3 LLM runs, diagnostic outputs were identical. KOA diagnostic agreement with physician A was 94.6% (105/111), and agreement for both diagnosis and affected side was 88.3% (98/111; Table 3). The baseline WOMAC total scores from the 3 LLM runs were highly correlated with the physician assessments (Pearson r=0.959-0.976; Figure 2A-C) and exhibited high consistency (ICC=0.956-0.973), with an MAE of 1.42 to 1.94 points and a mean signed difference of +0.81 to +1.04 points (Table 3). Scatterplots and Bland-Altman analyses showed that the 3 runs exhibited similar consistency characteristics (Figure 2D-F). The exact agreement rate between the 3 LLM runs and physician A’s baseline KL grading ranged from 80.2% (89/111) to 83.8% (93/111), and none of the grading discrepancies exceeded 1 grade. The linear-weighted κ coefficients ranged from 0.680 to 0.727, and the MAE in grading ranged from 0.162 to 0.198 (Table 3).

The mean actual WOMAC total score at 6 months was 20.89 points. The mean predicted scores from the 3 LLM runs ranged from 24.61 to 25.59 points, with an MAE of 5.92 to 6.58 points, an RMSE of 7.47 to 8.42 points, and a mean signed error of +3.72 to +4.70 points (Table 4). The LLM predictions showed a strong correlation with the observed values (r=0.779-0.814), and Bland-Altman analysis revealed a positive prediction bias in all 3 runs (Figure 3A-D). During actual follow-up, the WOMAC total score fluctuated in both directions, whereas the LLM clearly tended to predict higher scores (Figure 3E). The calibration slope of the LLM ranged from 0.748 to 0.778 (Figure 3F).

The MAE of the physician’s predictions was 3.86 points, which was lower than the 5.43 points for the persistence model (P=.001) and the 5.92 to 6.58 points for the LLM (P<.001; Table S3 in Multimedia Appendix 2). The mean signed error for the physician was +0.61 points, which was lower than the +3.72 to +4.70 points for the LLM (Table 4). Bland-Altman analysis showed minimal systematic bias between the physician’s predictions and observed values (Figure 3D). The calibration slope of physician predictions was 0.906, which was closer to the ideal value of 1 than the results from the 3 LLM runs (Figure 3F).

The LLM achieved ICCs of 0.982 and 0.965 for the baseline WOMAC total score assessment and the 6-month WOMAC total score prediction, respectively; however, the proportions of cases demonstrating complete agreement across the 3 runs were only 31.5% (35/111) and 6.3% (7/111; Table S4 in Multimedia Appendix 2). There was no significant difference in the distribution of baseline WOMAC total scores across the 3 LLM runs (P=.30; Figure 4A), whereas the overall difference in the 6-month predicted scores was significant (P<.001; Figure 4B), with the predicted scores from the first run being lower than those from the second and third runs (Table S4 in Multimedia Appendix 2). The median maximum difference in the 6-month WOMAC predicted scores across the 3 LLM runs was 2 (IQR 1-4; Figure 4C) points. The 3 LLM runs were completely consistent in KOA diagnosis and determination of the affected side. The rates of complete agreement for the baseline and 6-month predicted KL grades were 90.1% (100/111) and 85.6% (95/111), respectively, and the rates of exact agreement between any 2 runs were also high (Figure 4D).

Between-group differences in patient-level MAE were not statistically significant among the low, medium, and high WOMAC baseline tertiles (P=.64; Figure 5A). Both the MAE and signed error differed significantly across groups based on the direction of change in actual WOMAC total scores (P<.001 in all cases; Table S5 in Multimedia Appendix 2). The MAE and signed error were higher in the group with decreasing scores than in the group with increasing scores (Figure 5B and Figure 5C). An exploratory mixed-effects model showed that regular exercise, active weight control, and a decrease in the actual WOMAC total score were all associated with larger absolute prediction errors, with the regression coefficient for a decrease in the actual score being the largest (β=5.774, 95% CI 4.151-7.396; P<.001; Figure 5D).

Across the 3 LLM runs, exact agreement with the observed 6-month KL grade ranged from 66.7% (74/111) to 68.5% (76/111) compared with 86.5% (96/111) for the physician comparator and 85.6% (95/111) for the persistence model (Table 5). Across the 3 LLM runs, 98.2% (109/111) of the predictions differed by no more than 1 grade from the actual grading; misclassifications primarily involved underestimating KL grade 3 as grade 2, while misclassifications by the physician were relatively rare (Figure 6A-D). Both the physician and the persistence model had lower MAEs than the 3 LLM runs (Table S6 in Multimedia Appendix 2). Only 13.5% (15/111) of the patients showed progression in report-derived KL grade. Across methods, sensitivity ranged from 0% (0/15) to 20% (3/15), with no statistically significant between-method difference (P=.26). For LLM run 2, sensitivity was 20% (3/15; 95% CI 4.3%-48.1%). Specificity ranged from 80.2% (77/96) to 83.3% (80/96) for the LLM, compared with 99% (95/96) for the physician and 100% (96/96) for the persistence model (overall P<.001). Other classification metrics are shown in Figure 6E to 6H.

This study evaluated DeepSeek’s baseline assessment, 6-month outcome prediction, and operational stability in a prospective KOA cohort. The LLM showed high diagnostic agreement with a physician and performed well in WOMAC scoring and KL grading; however, its cross-sectional performance did not translate into reliable longitudinal predictions. This diagnostic agreement was assessed only in patients with established KOA and does not establish general diagnostic accuracy. The 6-month WOMAC prediction error and positive bias were higher than those of the physician comparator, and the model did not outperform the persistence model; its performance in KL grading and prediction of radiographic progression was also limited. Although interrun consistency was relatively high, absolute outputs still exhibited variability at the patient level.

The evaluated model’s strong baseline performance may reflect the structured information provided by standardized interviews and descriptive radiology reports. Previous studies have shown that LLMs can identify hip osteoarthritis or KOA using structured questionnaires, generate diagnoses consistent with those of specialist physicians using anonymized medical records, and extract imaging features and generate structured scores from free-text knee magnetic resonance imaging reports [9-11]. However, high correlations and ICCs alone do not establish clinically acceptable scoring accuracy. Absolute errors and Bland-Altman limits of agreement should also inform interpretation.

The evaluated LLM did not demonstrate added prognostic value over the persistence model, which assumes unchanged baseline status. Strong correlations with observed outcomes did not ensure accurate individual predictions, as reflected by the positive bias and poor calibration. Although DeepSeek demonstrated strong clinical decision-making capabilities in clinical case benchmarks [12], longitudinal prediction requires estimating state changes resulting from the combined effects of treatment, behavior, and disease fluctuations—dynamic information that is difficult to fully capture from a single baseline record. Existing research on medical LLMs has primarily focused on knowledge-based question-answering and diagnostic tasks, with limited evaluation of real-world longitudinal outcomes and calibration [13,14].

The evaluated DeepSeek configuration tended to predict symptom worsening, particularly when patients actually improved, suggesting limited capture of bidirectional short-term symptom changes. Pain and function in KOA can follow different trajectories and are influenced by exercise, weight management, and other interventions [21,22]. Regular exercise and weight control are associated with larger prediction errors, suggesting that an LLM based solely on a single baseline data point may struggle to capture subsequent symptom trajectories related to behavior changes.

Previous studies have also shown that LLMs’ performance in fine-grained KL grading remains limited [23,24]. In this study, the model was not provided with raw x-ray images but rather with descriptive imaging reports from which diagnostic conclusions had been removed. Baseline grading therefore involved extracting reported radiographic features, while longitudinal prediction targeted changes in report-derived KL grades. However, these findings do not establish performance in direct image-based progression prediction. Given the small number of progression events and class imbalance, overall grade agreement should be interpreted alongside progression sensitivity, specificity, and balanced accuracy.

Although ICCs were high across the 3 independent runs, exact agreement was low for both baseline WOMAC assessment and 6-month WOMAC prediction, with substantial interrun variability in some patients. The predicted values from the first run were lower than those from the subsequent 2 runs, suggesting that output fluctuations may include systematic run-level biases. Previous studies have also found that responses to the same questions are not entirely consistent across different time points or repeated runs [15,25]. High ICCs can coexist with run-dependent shifts in absolute scores even when patients remain similarly ranked. Such shifts could alter the estimated magnitude or direction of WOMAC change, limiting confidence in single-run predictions for individual patients.

The evaluated model may support clinical information extraction and preliminary baseline scoring under clinician supervision. Individualized prognosis requires anticipating future clinical changes, and its reliability cannot be inferred from strong baseline assessment performance. Previous studies have also shown that LLMs can produce KOA self-management materials and provide recommendations that are generally consistent with those of rehabilitation specialists [26,27]. In the future, unstructured information extracted by LLMs could be combined with raw imaging data, longitudinal symptom data, treatment adjustments, and behavioral trajectories to construct dynamic, multimodal, disease-specific models supervised by physicians.

This study has several limitations. This single-center study had only 15 progression events, limiting precision; progression-related findings should therefore be considered exploratory. Furthermore, the 6-month follow-up period and the KL grading system may not be sufficient to capture subtle structural changes. Different evaluators assigned baseline and follow-up report-derived KL grades without prior calibration or formal agreement assessment, potentially affecting progression classification and related prediction metrics. This study evaluated one DeepSeek configuration and one physician predictor, limiting generalizability to other models, physicians, populations, and clinical settings. Attrition may have introduced selection bias. The nonsignificant MCAR test did not establish that data were MCAR, and the IPW sensitivity analysis depended on assumptions about missingness and model specification. The relevant exploratory findings require validation in larger samples and external cohorts.

The evaluated DeepSeek configuration may support structured baseline KOA assessment, but this does not establish its suitability for individualized prognosis. Prognostic applications should demonstrate added value beyond a persistence model, adequate calibration, and stable patient-level predictions in independent cohorts. Further evaluation across clinical settings, model versions, and physician comparators is needed to define the clinical role of the evaluated DeepSeek configuration.