Data were derived from the 2023 Zhejiang Citizen Health Literacy and Tobacco Use Questionnaire administered by the Zhejiang Provincial Center for Disease Control and Prevention (March 2023-September 2023). The survey encompassed 56,340 respondents across 90 counties in 11 prefectural-level cities, collecting data on health literacy, tobacco use, and sociodemographic characteristics.

The study population comprised community-dwelling Chinese residents aged 15 to 69 years. We excluded individuals residing in institutional settings (military bases, hospitals, prisons, nursing homes, dormitories). The age range was selected to capture individuals who had completed compulsory education (≥15 y) and minimize potential cognitive impairment bias (<70 y).

This cross-sectional study was part of the 2023 China Health Literacy Survey, targeting noncollective Chinese residents aged 15 to 69 years in counties (cities and districts) across Zhejiang Province, excluding those living in military bases, hospitals, prisons, nursing homes, and dormitories. We excluded individuals younger than 15 years, as this age group typically has not completed compulsory education; we also excluded those older than 69 years, as this age group is more prone to cognitive impairment [26]. The external validation used data from 2024, which included a sample size of 56,863 individuals, also derived from the questionnaire survey on citizens' health literacy and tobacco use conducted by the Zhejiang Provincial Center for Disease Control and Prevention in Zhejiang Province [27]. During the data processing stage, no missing nor omitted entries were found, so there was no need for imputation of missing values. As a result, we obtained 56,340 complete and valid samples for the year 2023 and 56,863 complete and valid samples for the year 2024. The data acquisition rate was 100% [27].

The first part of the questionnaire assesses health literacy using the Chinese Health Literacy Scale developed by the Chinese Health Education Center using the Delphi method, comprising 50 items across 3 dimensions: knowledge and attitudes, behaviors and lifestyle, and health-related skills [28]. The overall Cronbach α for the scale was 0.95, and the Spearman-Brown coefficient was 0.94 [29].

The scale includes 3 question types: true/false (1 point for correct), single answer (1 point for choosing the correct one from multiple options), and multiple answers (2 points for selecting all correct answers from several options). If the health literacy score is ≥53, it implies that residents possess good health literacy (health literacy=1). Conversely, if the health literacy score is <53, it suggests that residents have limited health literacy (health literacy=0) [30].

We included a set of predictors based on previous studies. The set included sex (male or female), the continuous variable age (modeled as a continuous variable in all regression and machine learning analyses; the age brackets were quantile-based groupings for descriptive purposes only), ethnicity (Han or Minority), marital status (married or unmarried/divorced), education level (primary school and below, junior high school, senior high school or higher education including junior college, bachelor’s degrees, and postgraduate degrees) [31,32], occupation (technical/professional, student, manual, commercial, or unemployed/others), family size (1‐2, 3‐5, or >5), annual household income (0‐49,999, 50,000‐99,999, 100,000‐149,999, or ≥150,000), local registered residence (yes or no), current smoking (yes or no), presence of chronic diseases (yes or no), and self-rated health (very good, good, fair, bad, or very bad) [13,33-36]. The unemployment/other occupation group was set as the reference, and 4 dummy variables were established: technical/professional (Occupation_1), student (Occupation_2), manual (Occupation_3), and commercial (Occupation_4).

We used STATA 18.0, Scikit-Learn, and SHAP for gradient boosting decision tree (GBDT), extreme gradient boosting (XGBoost), logistic regression (LR), and random forest (RF) models in Python (version 3.8.10), and the “rmda” package for Nomogram in R (version 4.5.1) was used to perform statistical analyses. The nomogram was constructed based on LR. The 2 machine learning models, GBDT and XGBoost, excel at handling structured datasets and imbalanced datasets [37]. LR is a machine learning model mainly used for binary classification [38], while RF is a collection of decision trees that can reduce the adverse effects of overfitting [39]. In the 4 machine learning models, we randomly divided the data into a training set, validation set, and test set at an allocation ratio of 6:2:2. We also applied SMOTE to the training set. Meanwhile, we analyzed the performance indicators of the 4 models under the threshold of 0.1‐0.9 (Table S1 in Multimedia Appendix 1). Univariate and multivariable LR analyses were used to identify risk factors associated with health literacy.

GBDT is an ensemble learning algorithm that builds an ensemble of weak prediction models, typically decision trees, with the aim of improving the predictive accuracy of the model. GBDT is known for its ability to handle complex relationships and interactions between features, making it suitable for our dataset that has a large number of variables [40]. It is a widely used machine learning algorithm with high efficiency, accuracy, and interpretability. GBDT achieves state-of-the-art performance in many machine learning tasks such as multiclass classification, click prediction, and learning to sort [41].

XGBoost is an optimization of the gradient boosting framework that is designed to be highly efficient, flexible, and scalable. XGBoost differs from GBDT in its regularization approach, which helps prevent overfitting by adding a penalty term to the loss function for larger numbers of trees and deeper trees. This makes XGBoost particularly effective at handling large datasets with complex interactions while being efficient in terms of training time [42]. In addition, the advantages of XGBoost are that it can handle various types of data as well as missing values and outliers. It has the flexibility to choose different loss functions and weak predictive models, and it uses 10-fold cross-validation as a model evaluation strategy to avoid overfitting [43].

LR is the most commonly used simple machine learning model, mainly applied to binary classification problems. The core idea is to extend the linear regression model by using the sigmoid function to map the predicted values to the range of [0, 1]. LR has been widely applied in various fields of medicine, such as the prediction of cardiovascular diseases [44].

RF is an ensemble model composed of multiple decision trees. Each tree is grown randomly from the training set, and the final prediction is made by integrating the results of all the trees [45]. It can significantly reduce overfitting and increase prediction accuracy [46].

SHAP is a method for interpreting the output of any machine learning model. It is based on the concept of Shapley value in game theory and assigns a fair contribution score to each feature or feature value to indicate their effect on the model prediction. SHAP can not only help us to understand how the different features affect the model output and the magnitude of each feature’s contribution but also provide various visualizations of the global and local effects of features on the model [47].

A nomogram is a graphical representation that allows for the visualization of the relationship between multiple predictors and the outcome based on the results of a regression analysis. In our study, the data for 2023 were further divided into a development dataset and an internal validation dataset in a 7:3 ratio to create the nomogram model [48]. This division allowed us to develop the nomogram on the development dataset and validate its performance on the internal validation dataset, ensuring that the model’s predictive accuracy and generalizability were assessed [49]. A nomogram, as a visual model, can predict the probability of clinical outcomes and assist clinicians with making judgments [50].

Calibration curves were used to evaluate the performance of the nomogram model by comparing the predicted probabilities with the observed outcomes. This method provides a visual representation of the model’s calibration, with an ideal calibration curve being a diagonal line, indicating perfect agreement between predicted and observed probabilities [51]. Calibration curves are commonly used to evaluate models such as LR and Cox regression [52]. In this paper, we used it to assess the effectiveness of the nomogram model.

As shown in Table 1, to evaluate the performance of each machine learning model, we referred to and included the following indicators: area under the receiver operating characteristic (ROC) curve (AUC), sensitivity (equation 1), specificity (equation 2), accuracy (equation 3), positive predictive value (PPV; equation 6), negative predictive value (NPV; equation 7), and balanced accuracy (equation 5) [53,54]. Among them, balanced accuracy is the arithmetic mean of sensitivity and specificity [55]. Each machine learning model measured the probability that a resident had a high health literacy level. To further assess and interpret the predictive models, we calculated the net benefit of the nomogram model using decision curve analysis (DCA; equation 4). True positive (TP) and true negative (TN) indicate residents who were correctly identified as having a high level of health literacy or a low level of health literacy, respectively; false positive (FP) and false negative (FN) indicate residents who were inaccurately identified as having a high level of health literacy or a low level of health literacy, respectively. In binary classification, machine learning models predict the likelihood of individuals having health literacy. A threshold of 0.5 was used, classifying those with a probability of ≥0.5 as having a high level of health literacy and those with a probability <0.5 as not having a high level of health literacy.

To our best knowledge, this study conducted the first comprehensive approach combining nomograms and machine learning models to predict population health literacy, developed and externally validated using large-scale survey data. The models demonstrated robust performance across both development and external validation cohorts, establishing their utility for population-level health literacy prediction. The nomogram was constructed based on LR [56]. The AUC values of this nomogram were 0.821 in the internal validation model 1 and 0.814 in the external validation model 1, indicating its excellent predictive performance. Moreover, this nomogram is simple and easy to understand and has high interpretability in clinical practice.

In this study, the GBDT, XGBoost, LR, and RF models achieved AUC values exceeding 0.75 in both test and validation sets, indicating strong discriminatory power and low false prediction rates. To enhance interpretability and practicality, we constructed and validated 2 nomogram models. Both internally validated nomogram models and externally validated nomogram models exhibited diagnostic accuracy, as evidenced by their AUC values, and their calibration curves demonstrated good concordance between predicted and actual probabilities, signifying high precision at distinguishing individuals with differing levels of health literacy. SHAP analysis of the GBDT, XGBoost, LR, and RF models revealed 5 critical predictors for nomogram model 2. Despite varying variable compositions, models 1 and 2 exhibited strikingly similar performance, substantiating the viability of reducing the predictor set. In previous studies, nomograms combined with machine learning have been used to develop risk prediction models for cerebral infarction, eHealth literacy, and thymoma [57-59]. In conclusion, the results of machine learning and nomogram to predict diseases or other risks supported the feasibility and validity of the study hypothesis.

In this study, we further analyzed the performance indicators of the GBDT, XGBoost, LR, and RF models on the test set and validation set at the 0.5 threshold (Table 4) as well as the performance indicators of the 4 machine learning models on the validation set at the 0.1‐0.9 threshold (Table S1 in Multimedia Appendix 1). The health literacy prediction model demonstrated high efficacy for health literacy prediction, but at thresholds of 0.1 and 0.9, there were situations where the model’s sensitivity or specificity was low. Therefore, in practical applications, we should select an appropriate threshold to balance sensitivity and specificity, thereby achieving high accuracy and cost-effectiveness. As shown in Figure 7, when the threshold was 0.5, the LR model had a high net benefit and balanced sensitivity and specificity. The models constructed in this study contained 12 variables (with occupation as a categorical predictor represented by 4 dummy variables), which were more comprehensive than short health literacy screening tools such as the HL-SF3 [60] and HL-SF6 [61]. In the context of accelerating the construction of regional information health platforms and the digitization of health information in China [62], this model conducted population-based health literacy screening and monitoring based on big data. By integrating data from regional health information systems, this model can assess health literacy at the public health level, helping to identify health literacy gaps among different groups and regions. Moreover, the AUC values of model 1 and model 2 in the internal and external validations were both >0.75, indicating good predictive efficacy and the ability to provide a basis for policy formulation, resource allocation, and the implementation of targeted intervention measures, thereby improving overall health literacy levels.

To investigate the risk factors associated with health literacy, this study analyzed and integrated the findings from LR analysis and SHAP models. Overall education, age, occupation, household income, smoking, and self-rated health emerged as significant predictors related to health literacy. Studies have noted that education, age, and economic status are associated with health literacy [18,63]. Essentially, there was a strong positive relationship between educational attainment and health literacy, with individuals with higher education often having greater access to comprehensive health education and information. These individuals demonstrate enhanced abilities in understanding and processing such data, which can lead to a more effective grasp of medical advice, disease prevention measures, and the importance of health management [64]. Age has been identified as a key variable in health literacy, with a Beijing-based survey indicating higher health literacy among individuals aged 25 to 44 years [14]. Aging may enable the accumulation of health-related knowledge and skills through life experiences, but it can also be associated with cognitive decline, potentially affecting older adults’ ability to absorb and use health information [65]. Younger generations, on the other hand, may benefit from modern health education systems to gain access to updated health concepts and technologies [66]. Additionally, occupational disparities can influence health literacy. Professionals in technical or specialized fields often engage in specific training and continuous learning, which may include relevant health and safety knowledge and skill development [67]. For example, medical practitioners and educators, who benefit from advanced education, may also have access to favorable working environments and benefits such as health insurance and fitness facilities, factors that can contribute to their health literacy [68]. The association between household income and health literacy is well-documented. Individuals with higher economic status often have access to better educational resources and a wider range of in-depth health guidance, which are critical to enhancing their health literacy levels. Individuals who rate their health status positively may be more likely to actively monitor and manage their well-being, seeking out and applying health-related knowledge, which can result in higher levels of health literacy [13,69,70]. At the same time, there are studies that have shown that smoking habits are related to health literacy. Compared with smokers, nonsmokers have a higher level of health literacy [71]. Self-rated health is also related to health literacy [72]. Health literacy is defined as an individual’s ability to acquire, understand, and use health information and services and make appropriate health decisions [73]. Self-rated health is defined as an individual’s perception of their own health status [74]. Studies have shown that people who rate their own health as good have higher health literacy [72], while those who rate their health as average or poor have insufficient health literacy [75]. Meanwhile, the results of the LR and SHAP analyses showed that factors such as ethnicity were not significantly correlated with health literacy. Another cross-sectional study from Malaysia demonstrated that ethnicity was not significantly correlated with the level of health literacy [76].

In this study, the health literacy prediction model exhibited commendable performance, highlighting its substantial potential for effective implementation and wide-ranging practical application in addressing health literacy challenges. At the individual level, individuals predicted to have high health literacy can act as catalysts, not only imparting health knowledge and skills to their peers but also extending these teachings to those around them, including older adults, thereby enhancing their health literacy. As health literacy improves, it can boost work productivity [77], heighten health awareness, facilitate better self-management, and elevate overall quality of life. On a broader societal canvas, the health literacy prediction model can seamlessly interface with health big data systems in the future, allowing for a systematic and exhaustive assessment of health literacy rates across different population groups and regions [78]. This approach stands to reduce the reliance on large-scale questionnaire surveys, thereby saving costs and resources. This capacity facilitates targeted interventions and the strategic roll out of health education programs in populations or regions with low health literacy scores, progressively closing health literacy gaps and promoting a more equitable distribution of health care and education resources. At the national level, leveraging the health literacy prediction model provides policymakers with a unique vantage point to foresee the contours of the country’s health literacy landscape. It can direct strategic fund allocation to health literacy–deficient areas, concurrently orchestrating the refinement of health strategies and bolstering investments in cultivating a skilled workforce of health care technology professionals. At the same time, the reliability of the model can be verified through national medical data, and it can be incorporated into national primary health care work to monitor the health literacy levels of residents, thereby informing targeted measures to enhance health literacy levels. This concerted endeavor is geared toward a paramount objective: elevating the overall health literacy of the nation’s population, thereby charting a course toward comprehensive wellness and informed health care engagement.

In conclusion, the use of machine learning and nomograms for health literacy prediction can deepen the understanding of population health literacy and enable both individuals and policymakers to take proactive measures to improve public health and optimize resource allocation.

This study has several limitations. First, this study used cross-sectional data. Although our data showed correlations between variables, due to the nature of the data and the study design, we were unable to determine if these relationships were causal. Future studies may need to use a longitudinal research design or experimental methods to further explore the relationships between these variables. Second, although the sample size of this study was large enough, at 56,340 residents, the sample was concentrated in Zhejiang, China. The generalizability of the results may still be influenced by regional characteristics; this limits the model’s direct applicability to other provinces or cultural contexts. Future studies could use the same models to validate the findings in other provinces. Finally, as aforementioned, due to the limitations of data accessibility, this study included only basic information about the population in the model and did not include influencing factors such as dietary habits and the availability and utilization of health services. We included social factors such as education, occupation, and income to enhance the model’s accuracy. However, due to the limitations of the research method, it was impossible to capture dynamic changes in economic factors such as income. In the future, the design method can be changed, such as conducting a cohort study and incorporating more comprehensive data, to make the model’s predictions more accurate.

This study used machine learning and LR analysis models to successfully develop and validate a predictive model for assessing the health literacy level of Chinese adults. The models identified education level, age, annual household income, smoking, and occupation as the most influential determinants of health literacy. This predictive framework provides a practical tool for targeted intervention measures to enhance the health literacy of the population and can guide resource allocation and the formulation of targeted health promotion strategies.