This study was conducted as a web-based cross-sectional survey of health care professionals across the United Kingdom who use SystmOne EHRs (The Phoenix Partnership Ltd). The cross-sectional survey approach allowed the assessment of the applicability, internal consistency, and construct validity of the CLIQ framework. The web-based approach offered a convenient, affordable, and pragmatic way of conducting a study and collecting data. The choice of SystmOne was informed by its wide use across different health care settings in the United Kingdom, including general practices, urgent care centers, social care services, hospitals, and prison medical services to manage about 61 million EHRs [22]. CHERRIES (Checklist for Reporting Results of Internet E-Surveys) was used to guide the survey report [23].

Eligibility criteria for the study included being a health care professional with a clinical role and a regular user of SystmOne (defined as using SystmOne as part of routine professional activities to document clinical information). Administrative staff using SystmOne and health care professionals using SystmOne occasionally (eg, to check clinical records) were not eligible to participate in the study. Participants were invited through emails and social media platforms, including Facebook, LinkedIn, WhatsApp, Yammer, Digital Health Networks, and the Future National Health System Collaboration platform. Although there is no consensus about the adequate sample size needed to validate a questionnaire, the literature recommends recruiting at least 10 participants for each item of the scale being validated as a rule of thumb [24]. As there are 14 dimensions of the CLIQ framework, this was equivalent to 140 participants in our study.

Health care professionals were invited to use the CLIQ framework to evaluate the information quality of SystmOne. The survey questionnaire comprised the 14 items of the CLIQ framework and 2 questions about the occupation of the respondents to assess their eligibility (Multimedia Appendix 1). Each question was displayed on a separate mobile or computer screen to the participants. The responses were made mandatory to avoid missing data, which could limit the assessment of the construct validity [25]. Participants were able to change their answers using the back button. Bot detection and prevention of multiple submissions were turned on in Qualtrics (Qualtrics). An invitation containing a link to the participant information sheet, the consent form, and the questionnaire was shared with the health care professionals electronically through the channels described earlier. Two reminders, at least 2 weeks apart, were sent to encourage participation. Health care professionals were also encouraged to share the invitation with colleagues. Data collection took place between February 27 and June 7, 2022. The questionnaire was administered through a web-based survey platform, Qualtrics.

The survey result was downloaded from Qualtrics in an excel format and imported into SPSS (version 20; IBM Statistics). Only completed entries were analyzed as uncompleted entries stopped at the consent. SPSS was used to conduct descriptive statistical analyses and compute correlation coefficients and Cronbach α scores.

The 3 options for each dimension (eg, accurate, partly accurate, and not accurate) were recoded into the integers 1, 2, and 3, representing low, modest, and good quality, respectively. A descriptive statistical analysis was conducted and interpreted to demonstrate the applicability of the CLIQ framework. The distribution of the responses was expressed as frequencies, percentages, means, and SDs.

Spearman correlation coefficients were computed to investigate the linear relationship between the dimensions in the CLIQ framework. The ordinal nature of the data informed the choice of Spearman coefficients. Correlation coefficients of 0.1‐0.2 were regarded as poor, 0.3‐0.5 as fair, 0.6‐0.7 as moderate, and 0.8‐0.9 as very strong [26] .

The Cronbach α coefficient was computed to assess the internal consistency of the global scale (ie, CLIQ framework) and the subscales (ie, the informativeness, availability, and usability categories). A Cronbach α coefficient of 0.7 or above was regarded as an indication of the reliability of the scale, and an alpha coefficient between 0.6 and 0.7 was considered acceptable [26].

The SPSS data file was subsequently exported to SPSS Amos (version 28; IBM Statistics) to assess the construct validity of the CLIQ framework. Confirmatory factor analysis, a structural equation modeling technique, was used to assess the extent to which the survey data supported the construct validity of the CLIQ framework [27]. Confirmatory factor analysis was adopted because the CLIQ framework has multiple subscales (ie, informativeness, availability, and usability categories) that were predetermined [11].

The maximum likelihood estimation method was used for the confirmatory factor analysis. The model fit was assessed based on the Standardized Root Mean Square Residual (SRMR) and Comparative Fit Index (CFI) as recommended in the literature for studies with a sample size of less than 250 [28]. A CFI greater than 0.9 and an SRMR less than 0.08 indicate model fit [28].

Informed consent was obtained from each participant at the beginning of the web-based survey after they had been provided with the participant information sheet, containing detailed information about the study objectives, expectation of the participants, duties of the researchers, and relevant contacts. Participation was voluntary. A secure web-based survey platform was used for data collection as already described. No personal information that could be used to identify the participants was collected in the survey. Survey responses were anonymized with codes automatically assigned to participants by the survey platform. The results were securely stored in Imperial College Shared Drive. Participants could withdraw from the study at any time without giving any reasons. However, once the survey was submitted, the data could not be withdrawn as the responses were anonymous. Participants were not compensated for taking part. Ethical approval was obtained from the Imperial College research ethics committee (21IC7415).

A total of 109 health care professionals completed the survey, with two-thirds (67, 61.5%) being doctors and almost a quarter (26, 23.9%) being nurses or advanced nurse practitioners. The rest of the participants had other clinical occupations. Table 2 shows the distribution of the participants by occupation.

The mean quality score assigned to each dimension ranged from 2.2 for portability to 2.9 for security of clinical information in SystmOne (1, 2, and 3 indicate low, modest, and good quality, respectively). Most participants (97/109, 89%) ranked security of clinical information in SystmOne as good while only more than a third of the participants (42/109, 38.5%) ranked portability of clinical information in SystmOne as good. The summary of the assessment result is shown in Table 3.

There were positive correlations between all possible pairs of dimensions in the CLIQ framework (Multimedia Appendix 2). All the correlation coefficients were statistically significant (represented as asterisks) except for the correlations of portability with each of plausibility, relevance, trustworthiness, and security. There is a strong statistically significant correlation between conformance and consistency (Spearman ρ=0.751; P<.001).

The Cronbach α coefficients and corresponding 95% CIs for the informativeness, availability, and usability subscales were 0.78 (95% CI 0.70‐0.84), 0.69 (95% CI 0.58‐0.77), and 0.83 (95% CI 0.77‐0.88), respectively. Once security was removed from the availability subscale, the Cronbach α coefficient for the availability subscale increased marginally to 0.70 (95% CI 0.60‐0.78). The Cronbach α coefficient for the overall CLIQ framework is 0.89 (95% CI 0.85‐0.92).

Although the chi-square goodness-of-fit test (χ²_2.10=155.69; P<.001) did not demonstrate fitness of the model, CFI (0.86) and SRMR (0.08) suggest that the model fits modestly with the data. All the factor loadings (ie, covariance estimates) were positive and statistically significant, as shown in Table 4. Significant tests (SE, critical ratio, and P value) were not reported for the first item in each category (ie, accuracy, accessibility, and conformance) because their unstandardized covariance values were fixed as 1, which is a standard procedure in confirmatory factor analysis.

The statistical significance of positive standardized covariance estimates for all information quality dimensions supports the placement of the dimensions in their respective category. This is further illustrated in the path diagram shown in Figure 2. The high values of estimated covariances between the latent variables and the observed variables support the construct validity of the model. Still, the high covariance estimates between the latent variables (informativeness, availability, and usability) indicate overlap of the categories.

The study assessed the applicability, internal consistency, and construct validity of the CLIQ framework based on the pilot CLIQ assessment of SystmOne EHRs. Overall, the CLIQ dimensions had good quality scores except for portability which had a modest score. The inter-item correlations were all positive and not likely due to chance. The Cronbach α score demonstrated a good internal consistency of the CLIQ framework and its informativeness, availability, and usability subscales. The results of the Confirmatory Factor Analysis provided a modest support for the construct validity of the CLIQ framework.

The combination of good security and modest portability of clinical information in SystmOne indicates the possibility of a trade-off between security and portability. Portability might have been limited inadvertently to improve the security of clinical information. Similar trade-offs, such as between accessibility and security, have been documented in the literature [29]. Software developers need to be vigilant to identify potential trade-offs that may compromise the quality and safety of care.

The positive and mostly significant correlations between the items in the scale indicated a close relationship between the dimensions [30]. This was not unexpected because all dimensions are components of the same CLIQ framework. The generally low values of the correlations demonstrated that the dimensions, although related, are distinct [30]. The high correlation between conformance and consistency of presentation is understandable, as both concern presentation of information. Similarly, the high covariance between the latent variables (informativeness, availability, and usability), which indicate that the categories are not entirely distinct, is not strange in information quality research. Categories in information quality frameworks are not mutually exclusive as each dimension in an information framework often fits into multiple categories [31]. However, there is a scope for model revision to further explore this overlap in future CLIQ research.

The overall good internal consistency of the CLIQ framework and its subscales (ie, informativeness, availability, and usability) indicates the reliability of the CLIQ framework [32]. Although the study was unable to rely on fit indices primarily influenced by sample size such as the chi-square goodness-of-fit index [23], the index least affected by sample size, the SRMR [28], supports the model fit.

This study used similar methods as a study on the validation of the Modified Enlight Suite (MES), a generic mHealth assessment questionnaire [25]. The MES study demonstrated an overall good internal consistency and modest construct validity of the MES based on a survey of more than a thousand medical students and health care professionals who assessed a freely downloadable COVID-19 app in Ireland [25]. The differences in the uptakes of the 2 studies are probably related to the choice of DHTs and study population. This study population was limited to health care professionals using SystmOne. In contrast, the MES study was open to all health care professionals and medical students who could download the Irish COVID-19 app.

To the best of our knowledge, the CLIQ framework is the first information quality framework for DHTs, of which the internal reliability and construct validity have been assessed. Only the face and content validity of most existing information quality frameworks were assessed [11]. This study went a step further to explore the internal consistency and construct validity of the CLIQ framework.

Although the choice of SystmOne EHRs ensured participation of a multidisciplinary population of health care professionals in the testing of the CLIQ framework, allied health professionals were less represented. In addition, the voluntary nature of recruitment established a potential for systemic bias because those who were engaged in digital solutions were more likely to respond. The use of mandatory responses, although useful for survey data completeness, could introduce bias if participants provided less thoughtful responses just to complete the survey. Finally, the low response rate limited the assessment of the construct validity of the CLIQ framework [24]. The low uptake was probably due to the busy schedules of the health care professionals at a time when the National Health System was under immense pressure due to COVID-19 pandemic. Nevertheless, the sample size was sufficient to assess the CLIQ framework’s applicability and reliability as demonstrated by the narrow CIs for the Cronbach α coefficients. Future studies on the validity of the CLIQ framework need to consider and address these limitations.

The CLIQ research highlights the importance of information quality and its relevance to the quality and safety of care. The CLIQ framework demonstrated a high reliability and a modest construct validity. The CLIQ framework offers a pragmatic approach to assessing the quality of clinical information in DHTs and could be applied as part of information quality assurance systems in health care settings to improve quality of health information.