In the past decade, 1 of the most challenging components of technology development has been to ensure the usability of tools to ensure their quality in use [1,2]. To support use of the technology, usability must be assessed during the development process [3]. Usability is the measure of the quality of a user’s experience when interacting with a system—whether a website, mobile technology, or any user-operated device [4]. In other words, usability refers to how well users can navigate a system to achieve their goals and how satisfied they are with the process. A successful system needs to work for its users, and it needs to work well. However, many mHealth technologies have been made available to the public, with insufficient attention devoted to their design, development, and evaluation [5]. Technologies produced with poor design and inadequate consideration of the needs of their intended users will be difficult to learn, misused, or underutilized and will ultimately fail to accomplish their objectives [6]. For this reason, usability has been widely recognized as a critical consideration in evaluating the efficacy of technologies [7].

Usability is especially critical for mobile technology, which is widely used in health care [8-10]. In fact, there were more than 800,000 mobile health (mHealth) apps in Apple App Store and Google Play Store in 2020. There is continued growth, with about 200 mHealth apps added each day, with some focusing on healthy eating, physical activity, and improved mental health [11-13]. To date, several studies have evaluated the effect of mHealth apps on disease management and prevention [14-17].

Research to evaluate the usability of mHealth apps has been conducted for various health states, such as chronic obstructive pulmonary disease, HIV, obesity, depression, anxiety, dysmenorrhea, and premenstrual syndrome [18-26]. However, to ensure the rigor of mHealth technologies, it is necessary to create tools to evaluate the quality and usability of mHealth apps [27].

The quality of mHealth apps requires understanding the context of their use and ensuring the apps’ usability [28]. Therefore, it is important to develop mHealth tools using rigorous usability evaluation tools. Although there are several mobile app assessment tools, most have a large number of items or have only been validated for a single disease or in a single language [29,30]. Given these limitations, there is a need for instruments that are validated across languages and diseases. This is especially true in South Korea, which has 1 of the highest rates (94%) of smartphone use in the world. This high penetration of smartphones has enabled the rapid integration of mHealth apps [31,32]. However, despite the high usage of mHealth apps in South Korea, there is a dearth of availability of simple instruments to assess their usability.

This study sought to translate the Health Information Technology Usability Evaluation Scale (Health-ITUES) from English into Korean and validate its use. The Health-ITUES is a customized questionnaire comprising 20 items and a modified version of the Health Information Technology Usability Evaluation Model (Health-ITUEM) [33]. The English version of the Health-ITUES has been previously validated through exploratory factor analysis (EFA) and confirmatory factor analysis after use by nurses [34] and community-dwelling adults with HIV [35]; however, it has not been translated into other languages and validated. This study translated the Health-ITUES into Korean and validated it in a sample of 244 women who experienced menstrual-related symptoms and used a menstrual tracking app called PINKDIARY.

This study was approved by the institutional review board of the Catholic University of Pusan (CUPIRB-2019-003) before the commencement of study activities. Inclusion criteria for this study were unmarried women >20 and <39 years of age who were previously or are currently using the menstrual-tracking app PINKDIARY for more than a month. The app records the highest usage rate in Korea, and as of October 2021, it was ranked sixth in the health and fitness category in Apple App Store. This app is the official app of the Korean Association of Obstetricians and Gynecologists. PINKDIARY is used to track menstruation and premenstrual symptoms. Features of the app include symptom records, doctor consultations, an online community, and a shopping mall for menstrual items (eg, pads, tampons, and menstrual cups). Marriage and age were also inclusion criteria, as these may have affected the participants’ usage of or experience with using the app.

The sample size was set to ensure a minimum number of participants based on the number of items in the instrument. Nunnally [36] recommended a minimum participant ratio of 10 participants:1 survey item. In this study, the target sample size was between 200 and 250 after multiplying the number of questions (20) by 10. This estimate was based on an anticipated attrition rate of about 20%. In the past, the dropout rate in app-related studies was about 20%-50%. Since this study was not an intervention study, the dropout rate was estimated at 20% [37].

Recruitment was conducted through the KakaoTalk (Kakao Corp) messenger and other online communities (eg, Everytime). Potential participants sent a screenshot of the PINKDIARY app to the researcher’s messenger to authenticate their use of the app. The online consent form and questionnaire were developed in Survey Monkey, and the link was sent to participants on the KakaoTalk messenger. After filling in the consent form and questionnaire, a 2000 won (about US $2) online coffee coupon was sent to the participants as a token of appreciation for their time.

We customized the Health-ITUES, which was previously validated in a sample of persons with HIV, for women with menstrual discomfort [25]. In this study, menstrual discomfort was defined as primary dysmenorrhea and premenstrual syndrome (PMS), which are the most common menstrual discomfort symptoms among women of reproductive age [38,39]. The modified version was reviewed by the senior author of this manuscript, RS.

The translation and back-translation followed WHO guidelines [40].

Descriptive statistics were used to analyze the demographic characteristics of the study participants. Reliability and validity were analyzed using Cronbach α, Pearson correlation coefficient, and EFA. All analyses used IBM SPSS Statistics 24.0. Two-sided P values of <0.05 were considered statistically significant.

All study participants provided informed consent, and the study design was approved by the appropriate ethics review board.

Five women completed the draft version of the survey and provided the following feedback about the Health-ITUES:

As a result of this feedback, item 8 was changed. More specifically, further explanation was added to clarify the term “equitable process” as follows: “Health equity refers to health equality, which means ensuring the optimal level of health for all people, regardless of income and educational level.”

A total of 244 unmarried female participants completed the Health-ITUES. The participants’ ages ranged from 20 to 36 years (mean=22.45, SD 3.039). Meanwhile, 127 (51.2%) participants majored in a health-related topic, and 117 (47.0%) participants majored in non-health-related topics for their university degrees. Participants resided in the following cities and provinces: Seoul (n=76, 30.5%), Gyeonggi-do (n=61, 24.6%), Chungcheong-do (n=52, 21.7%), Gyeongsang-do (n=48, 19.4%), and Else (7, 2.8%).

Internal consistency reliability was measured by Cronbach α, and the results are presented in Table 3. All items showed good Cronbach α values (>0.8) ranging from 0.83 to 0.94 (Table 3). All values were less than 0.95, which demonstrates that there was no redundancy among the items [44]. Internal scale correlation between items ranged from 0.45 to 0.71, indicating moderate to strong correlations. Notably, perceived usefulness was more highly correlated with impact than other subscales.

A major challenge to technology development is ensuring the usability of the tools [1-3]. However, many mHealth tools are currently available with little attention to their usability [5]. Technologies produced with poor design and inadequate consideration of the needs of their intended users are often difficult to use, and the consumers often cannot accomplish their goals, as a result [6]. Therefore, usability has been widely recognized as an important factor in the development of technology [7].

To address the need for understanding the usability of technology, various studies have been conducted [29,30]. However, each assessment tool could not be easily used, because it had too many questions, took a long time to answer due to difficult questions, was developed for specific users, or was developed only in English [23,29,30,45]. Therefore, the Health-ITUES, a simple and verified tool for multiple populations, was chosen for translation and validation. Because the Health-ITUEM, which is the basis of the Health-ITUES, was developed based on several usability models with strong reliability and validity, including the technology acceptance model (TAM) and ISO9241-11 [46-48]. Additionally, the Health-ITUES was validated in a sample of nurses and community-dwelling adults with HIV [33-35].

This study sought to not only translate but also verify the Health-ITUES considering the Korean context. In step 1, the authors modified the Health-ITUES that was used to evaluate an app for adults with HIV to an app for women suffering from menstrual discomfort. In Korea, there are fewer HIV-infected people compared to the United States. Additionally, because of negative views regarding homosexual contact, HIV-infected people are more likely to not disclose themselves [49]. As of October 2021, when searching for HIV/AIDS-related apps in Apple App Store and Google Play Store, it was difficult to find mHealth for patients with HIV in Korean. To obtain sufficient app users for the validation of the translated tool, the Health-ITUES was modified to enable usability evaluation of the menstrual app PINKDIARY for managing dysmenorrhea experienced by 75% of domestic women [38,50].

In step 2, the Health-ITUES was translated based on WHO guidelines [40]. These guidelines have been used to translate various instruments into many languages, such as Japanese and Arabic, and are not limited to Korean [51-53]. The process of translation and adaptation of instruments were as follows: forward translation, back-translation, and cognitive testing. In this study, 3 experts who majored in nursing or medical informatics and 1 professional English/Korean translator participated in the forward translation and back-translation, respectively. The 2 translators independently translated and reconciled any discrepancies after the forward translation following WHO guidelines. This study was further strengthened by the review of the Korean Health-ITUES by 1 of the authors, who validated the Health-ITUES for mHealth technology. This process ensured that the original meanings of the Health-ITUES items were retained.

Another strength of this reliability and validity is our study sample size. The sample size exceeded the number of items, 20 × 10, and the participants live in various regions in Korea. Their field of work (major) is also not biased, so it can be said that geographical biases are small.

Moreover, in step 3, internal scale correlation ranged from 0.45 to 0.71, which indicates moderate to strong correlations. Notably, perceived usefulness was more highly correlated with impact than the other factors. The results mirrored the findings from the validation study using the Health-ITUES in a sample of adults with HIV [34]. In EFA, in the case of values for usefulness, 5 items (“I think PINKDIARY provides a more equitable process for the self-management of my menstrual-related symptoms.”) were included as 3 components because the values were so similar: 0.416, 0.410, and 0.414. As a result, we decided to keep this item with the first component perceived usefulness with which it is most closely conceptually aligned [54].

One limitation of this study is that when recruiting survey participants, only the experience of using the menstruation-related mHealth app was checked and no restrictions were placed on the past period of use. Future research should recruit participants by suggesting clear past usage periods based on objective evidence. However, this study successfully translated the Health-ITUES from English into Korean and validated it, and this instrument can be used to evaluate the usability of mHealth apps. These findings will contribute to the systematic evaluation of the rapidly growing field of mHealth apps.

The Health-ITUES demonstrated reliability and validity for use in assessing mHealth apps’ usability in young Korean women with menstrual discomfort. Given the strong psychometric properties of this tool in Korean and English and across 2 different health conditions, the Health-ITUES is a strong instrument for evaluating the usability of mHealth apps.