In this study, we conducted a retrospective analysis of newly referred patients to the rheumatology department at Frisius Medical Center, Leeuwarden, the Netherlands (the pipeline is shown in Figure 1). We developed and validated machine learning prediction models to assess the FRYQ’s ability to differentiate inflammatory from noninflammatory rheumatic conditions using only preconsultation patient responses. Multimedia Appendix 1 provides the Dutch version of the questionnaire, and Multimedia Appendix 2 provides the translated version in English.

We used two independent cohorts: (1) a development dataset (dataset A), which comprised 728 adult patients who completed the FRYQ between August 2017 and June 2019 as part of routine care, and (2) an external validation dataset (dataset B), which comprised 373 adult patients from the Joint Pain Assessment Scoring Tool (JPAST) study cohort [12] visiting the clinic between January 2023 and March 2024 who completed the FRYQ as an additional research instrument.

The FRYQ was developed by the rheumatology clinic at the Frisius Medical Center to assist in prioritizing newly referred patients. The data in dataset A were gathered as part of standard clinical care. Informed consent for the use of anonymized data in research was secured through the opt-out procedure, whereby all patient data were eligible for inclusion unless an individual filed a formal objection. According to Article 1(b) of the Dutch Medical Research Involving Human Subjects Act (Wet medisch-wetenschappelijk onderzoek met mensen [WMO]), ethical review is required only for studies that aim to obtain medical-scientific knowledge and in which participants are subjected to an intervention or are required to follow a behavioral regime [13]. As the FRYQ questionnaire was administered as part of routine care and did not involve any intervention or imposed behavior, the study does not fall under the scope of the WMO and therefore did not require approval by a medical ethics review committee.

In contrast, dataset B consisted of patients enrolled in the JPAST study, which involved time-consuming study procedures. Therefore, the use of the FRYQ in this dataset was reviewed and approved by the medical ethics committee of the Frisius Medical Center (reference number RTPO-1104) as part of the JPAST study protocol. Since participation in the JPAST study involved procedures beyond standard care, each participant provided written informed consent authorizing the use of their data for research.

For both datasets, patient data security was ensured by assigning each patient a unique sequential identifier through pseudonymization. Data were stored and analyzed in a secure, password-protected database accessible only to authorized personnel. Participants retained the right to opt out of the use of their data for research purposes and did not receive financial compensation for their involvement in this study. All study procedures were conducted in accordance with the principles of the Declaration of Helsinki.

The primary input variables consisted of responses to the 67 closed-ended questions from the FRYQ supplemented by patient age and sex. The FRYQ is an 87-item tool (including 20 open-ended questions not used in this analysis) designed by rheumatologists for triaging rheumatology patients. Question topics covered pain characteristics and patterns, stiffness symptoms and duration, functional limitations, associated symptoms, comorbid conditions, medication responses, and patient behaviors and characteristics. No laboratory test results or imaging data were included as predictors in the model.

Patients with incomplete demographic and questionnaire data were removed from dataset A (127/862, 14.7%) and dataset B (166/552, 30.1%; Figure S1 in Multimedia Appendix 3). Furthermore, we had some patients with inconsistent diagnostic labels removed from dataset A (7/862, 0.8%) and a few with missing diagnosis information removed from dataset B (36/552, 6.5%).

We conducted threshold analysis to identify an optimal probability cutoff for distinguishing IRD from non-IRD cases in clinical practice, evaluating sensitivity, specificity, positive predictive value, and negative predictive value at various thresholds.

To understand feature importance and explain model predictions, we conducted Shapley additive explanation (SHAP) analysis [17]. This approach quantified the contribution of each question to the model output, allowing us to identify which questions were most informative for detecting inflammatory conditions.

Dataset A comprised 728 patients, of whom 239 (32.8%) were male and 489 (67.2%) were female (Table 1). Nearly half (257/582, 44.2%) of the patients in the training dataset had IRDs. The most common disorders associated with IRD were RA (62/582, 10.7%), gout (45/582, 7.7%), and spondyloarthritis (43/582, 7.4%). Among noninflammatory arthropathies, the most frequent were osteoarthritis (119/582, 20.4%), tendinopathy (92/582, 15.8%), and fibromyalgia (91/582, 15.6%).

The external test dataset B included 350 patients with complete information, comprising 106 (30.3%) male and 244 (69.7%) female individuals. Only 26.6% (93/350) of the patients were diagnosed with an IRD, which was notably lower than those in the training and validation partitions of dataset A (257/582, 44.2% and 67/146, 45.9%, respectively). In contrast, the JPAST data contained a higher proportion of patients with an “other” diagnosis, indicating that this dataset represented a somewhat different patient population.

After hyperparameter optimization (α=.07; L1-ratio=0.16; Figures S2 and S3 in Multimedia Appendix 3) and feature selection, our model reduced the 67 closed-ended questions to 28 key items (Table S1 in Multimedia Appendix 3). This streamlined model effectively distinguished IRD from non-IRD across multiple validation steps.

In the development dataset A, the reduced model achieved an AUC-ROC of 0.70 (95% CI 0.67-0.74) in the training set and 0.71 (95% CI 0.64-0.78) in the hold-out validation set, performing comparably to the full model (AUC-ROC of 0.67, 95% CI 0.63-0.72 and 0.72, 95% CI 0.64-0.78, respectively; Figure 2A).

Importantly, when tested in the independent dataset B, the model demonstrated robust generalizability, with an AUC-ROC of 0.72 (95% CI 0.67-0.78) for the reduced model and 0.68 (95% CI 0.63-0.75) for the full model (Figure 2B), confirming the model’s stability across different patient populations.

The FRYQ score (model-predicted probability) consistently distinguished IRD patients from non-IRD patients across both datasets (P<.001; Figure 3). In the development set A, the median FRYQ score was 0.47 (IQR 0.31-0.65) for patients with IRD, versus 0.30 (IQR 0.16-0.42) for patients without IRD. Similarly, in replication set B, the median score was 0.46 (IQR 0.26-0.66) for patients with IRD, compared to 0.24 (IQR 0.13-0.39) for patients without IRD. When examining specific conditions, most IRDs exhibited higher scores than non-IRDs, with 2 notable exceptions: spondyloarthritis consistently scored lower among IRDs, whereas tendinopathy scored relatively high (>0.35) among non-IRDs, suggesting areas where diagnostic discrimination was more challenging. Fibromyalgia consistently showed very low scores (<0.2), making it a promising target for specialized detection.

The key features for identifying an IRD in the newly referred population (Figure 2C) were questions related to duration of concerns (questions 1.10 and 2.8), male sex, and decrease of pain upon movement (question 1.4). Moreover, the presence of certain comorbidities influenced the model’s assessment. Ulcerative colitis and Crohn disease (question 4.2) showed a positive association with IRDs, whereas chest pain (question 4.12) showed a negative association. Additionally, respondents diagnosed with an IRD more frequently reported that nonsteroidal anti‐inflammatory drugs such as ibuprofen and diclofenac were effective in managing their symptoms (questions 2.9 and 1.11).

Next, we defined the optimal clinical threshold for case selection. On the basis of the training data, we set the cutoff at 0.30 (Figure S4 in Multimedia Appendix 3) as it captured more than three-fourths of IRD cases (sensitivity=0.76) while ensuring that half of the cases without IRDs were excluded (specificity=0.49). This threshold showed similar sensitivity (0.71) and specificity (0.56) in the external validation dataset. The positive predictive value was lower in the external validation dataset (0.39 vs 0.56 in dataset A), attributable to the lower IRD prevalence (93/350, 26.6% vs 257/582, 44.2%; Table S4 in Multimedia Appendix 3).

The calibration curve indicated that the model slightly overestimated IRD probability in the external test dataset (Figure S5 in Multimedia Appendix 3), reflecting the difference in disease prevalence between the training and validation populations. Despite this, the discriminatory ability remained strong, with IRDs consistently receiving higher scores than non-IRDs across both datasets, reinforcing the FRYQ’s potential value as a triage tool.

Our analysis extended beyond distinguishing inflammatory from noninflammatory conditions to evaluate the FRYQ’s ability to identify specific diagnoses. Patients with fibromyalgia showed particularly distinctive response patterns compared to those with inflammatory conditions (Figure 3), making this a compelling diagnostic target with sufficient representation in our datasets.

The fibromyalgia-specific model demonstrated strong discriminative ability, achieving an AUC-ROC of 0.85 (95% CI 0.78-0.90) in the development dataset and maintaining robust performance in the external validation dataset with an AUC-ROC of 0.81 (95% CI 0.75-0.86; Figure 4). SHAP analysis identified key predictive features for fibromyalgia: female sex, younger to middle age, fatigue (question 6.3), chest pain (question 4.12), extended symptom duration (questions 1.10 and 2.8), and waking up tired (question 5.9). The model effectively differentiated fibromyalgia from RA, although some misclassification occurred with noninflammatory conditions such as hypermobility (Figure 6A in Multimedia Appendix 3).

For RA—a critical target for early rheumatology intervention—our model showed moderate discrimination, with AUC-ROC values of 0.74 (95% CI 0.63-0.83) in the development set and 0.77 (95% CI 0.66-0.86) in the external validation set (Figure 5). Key predictors included shorter disease duration (questions 1.10 and 2.8), pain improvement with movement (question 1.4), recent severe pain (question 6.1), onset at middle age or later, and fewer respiratory or chest symptoms (questions 4.13 and 4.12) or hip concerns (question 1.9). The model’s primary challenge was differentiating RA from other IRDs with similar presentations, such as reactive arthritis, polymyalgia rheumatica, and gout (Figure 6B in Multimedia Appendix 3).

Our results demonstrate that the FRYQ can effectively distinguish between IRDs and noninflammatory musculoskeletal concerns before patients’ first rheumatology consultation. The model achieved robust performance, with an AUC-ROC of 0.71 in the development dataset and 0.72 in the external validation dataset, indicating good generalizability. Importantly, we found that a reduced set of just 28 questions from the original 67 closed-ended items maintained equivalent discriminative power, suggesting potential for a more efficient screening tool.

Beyond the primary classification task, our analysis revealed that the FRYQ demonstrates even stronger performance in identifying specific conditions, particularly fibromyalgia (AUC-ROC of 0.85 and 0.81 in the development and replication datasets, respectively). The FRYQ also showed adequate ability to detect RA cases (AUC-ROC of 0.77 in the external validation dataset), although it had more difficulty distinguishing RA from other inflammatory conditions.

These findings have important clinical applications. In settings with limited rheumatology resources, the FRYQ could serve as a triage tool to streamline the large influx of referrals (overreferral) by prioritizing the referred patients with higher probability of inflammatory disease for early specialist intervention. This could result in more efficient allocation of specialist resources, potentially reducing waiting lists for a substantial number of patients with IRD. In light of the anticipated shortage of clinical professionals, it is increasingly important to identify which patients truly need rheumatologists’ care versus those who can be managed elsewhere [18].

For example, the FRYQ’s ability to identify fibromyalgia among the rheumatology waiting list could help redirect these patients to alternative care providers instead (pain specialists, physiotherapists, or dedicated programs), freeing capacity at the clinic for patients needing rheumatological care. Currently, rheumatologists spend substantial time on patients they cannot effectively treat as approximately 60% of rheumatology referrals involve noninflammatory conditions (such as fibromyalgia) [10]. This optimized allocation may also offer an additional benefit: the extra capacity created could reduce primary care physicians’ reluctance to refer clinically ambiguous cases that would benefit from specialist input.

Finally, the FRYQ could aid clinicians in their decision to request extra laboratory workup. When responses suggest RA, clinicians could order relevant serological markers (anticitrullinated protein antibody, rheumatoid factor, C-reactive protein, and erythrocyte sedimentation rate) before the initial visit, potentially streamlining the diagnostic process and reducing time to treatment for conditions requiring early intervention [4-7].

The clinical threshold of 0.30 that we identified balances sensitivity (capturing 71%-76% of inflammatory cases) with reasonable specificity (49%-56%), making it suitable for initial screening where missing IRD cases would be more concerning than false positives. As reported by Graydon and Thompson [19], primary care physicians and rheumatologists frequently disagree on referral urgency (47%), with rheumatologists upgrading 17% of referrals to urgent status—cases initially overlooked by GPs. This underlines the potential value of applying such a threshold in practice, although adjustments may be required depending on local context, such as disease prevalence and health care setting.

Our findings align with the growing recognition that structured questionnaires can support diagnostic decision-making in rheumatology by systematically capturing and integrating patients’ own experiences and symptom descriptions. While over 100 symptom checkers currently exist [20], their adoption has remained limited due to lack of validation or suboptimal diagnostic accuracy [20,21]. Therefore, the availability of a validated tool such as the FRYQ may fulfill a need in the field. As far as we know, this is the first questionnaire developed in the Netherlands for triaging musculoskeletal concerns that has been validated using real-world patient data.

The predictive factors identified in our SHAP analysis align with established clinical knowledge. Pain improvement with movement, shorter morning stiffness duration, and good response to nonsteroidal anti-inflammatory drugs all emerged as important indicators of inflammatory disease [22-24]. The association between inflammatory bowel disease and IRDs supports known connections between these conditions [25]. For fibromyalgia, we observe that patients report sleep problems, persistent pain, and fatigue [26].

Beyond questionnaires, there have been other digital approaches proposed in the literature for improving rheumatology triage. Natural language processing (NLP) of referral letters has shown promising results in studies from the United Kingdom [27] and the Netherlands [28], potentially complementing questionnaire-based methods. Similarly, patient-reported outcomes have been investigated, although Schäfer et al [29] found these to be less effective at distinguishing inflammatory from noninflammatory rheumatic conditions.

Several limitations of this study should be acknowledged. First, while we validated our model in an external dataset, both cohorts originated from the same medical center, potentially limiting generalizability to different health care settings or patient populations. Second, we observed a rather big difference in prevalence of inflammatory conditions (257/582, 44.2% vs 93/350, 26.6%). This affected the model’s calibration in external validation, suggesting the need for recalibration when applying to populations with different disease prevalence.

These prevalence differences likely stem from distinct patient selection mechanisms. Dataset A included patients with prescheduled outpatient appointments who completed the questionnaire during their first visit as part of routine care in Medical Center Leeuwarden (approximately 70% of referrals). In contrast, dataset B comprised volunteers from the JPAST study (approximately 40% of referrals). The time-consuming study procedures of the JPAST study likely led to an overrepresentation of nonurgent (non-IRD) cases among the volunteers.

We should also acknowledge that our model showed moderate discriminative ability (AUC of approximately 0.72) rather than excellent performance. Therefore, it could not reliably distinguish certain conditions such as spondyloarthritis from noninflammatory diseases. Rare inflammatory conditions were also underrepresented.

Crucially, our study was tailored to patients already referred to rheumatologists with the goal of optimizing the time from referral to specialist consultation. This means that the FRYQ cannot be used as a prescreening tool at GP offices. Importantly, it was not designed to exclude patients from care but rather to guide referred patients to the most appropriate care providers and streamline referral pathways.

To definitively establish the FRYQ’s clinical utility, we propose conducting a randomized clinical trial in the future to assess whether FRYQ-guided triage leads to faster appropriate treatment, reduced unnecessary appointments, and improved patient satisfaction. Furthermore, our study would benefit from additional replication in other countries to assess how health care system differences affect its effectiveness. Referral rates and treatment protocols vary substantially between countries—for instance, 21% of patients with musculoskeletal conditions are referred to specialists in Spain compared to 11% in the Netherlands and 5% in Sweden [30].

In the future, we also plan to incorporate NLP techniques to leverage open-ended questions as they may capture nuanced symptom descriptions that structured items tend to miss. Modern NLP methods, including large language models, show potential for supporting triage [31] and the identification of rare or unrecognized diseases using free-text data [32,33]. For spondyloarthritis in particular, large language models could be prompted with educational information on axial spondyloarthritis recognition to improve detection.

Finally, we may enhance the questionnaire by incorporating questions on occupational stress given that it is a known predictor of non-IRD [34,35]. Further refinement could also involve integrating FRYQ scores with serological markers and imaging to create a comprehensive multimodal prediction model. This multimodal approach might overcome some limitations of questionnaire-based triage alone, particularly for discerning conditions with overlapping symptomatology (ie, overlapping symptoms of osteoarthritis and early RA [36]).

The FRYQ shows promise in distinguishing inflammatory from noninflammatory rheumatic conditions in newly referred patients, particularly fibromyalgia and RA, before specialist assessment. While not a substitute for clinical judgment, this tool could help optimize the time from referral to specialist consultation by prioritizing cases likely to involve IRDs and guiding others toward alternative care pathways or additional laboratory testing. These potential benefits merit evaluation in prospective implementation studies, especially given the projected shortage of rheumatology professionals.