This paper is in the following e-collection/theme issue:

Decision Support for Health Professionals (1147) Research Instruments, Questionnaires, and Tools (545) Usability of Apps and User Perceptions of mHealth (902) Quality Evaluation and Descriptive Analysis/Reviews of Multiple Existing Mobile Apps (363)

Published on in Vol 10, No 1 (2022): January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/28621, first published .
Validity and Reliability of the Korean Version of the Health Information Technology Usability Evaluation Scale: Psychometric Evaluation

Validity and Reliability of the Korean Version of the Health Information Technology Usability Evaluation Scale: Psychometric Evaluation

Validity and Reliability of the Korean Version of the Health Information Technology Usability Evaluation Scale: Psychometric Evaluation

Authors of this article:

Jisan Lee1, 2 Author Orcid Image ;   Rebecca Schnall3, 4 Author Orcid Image
Article Authors Cited by (4) Tweetations (3) Metrics

    Original Paper

    1Department of Nursing Science, College of Life & Health Sciences, Hoseo University, Asan, Republic of Korea

    2The Research Institute for Basic Sciences, Hoseo University, Asan, Republic of Korea

    3School of Nursing, Columbia University, New York, NY, United States

    4Department of Population and Family Health, Mailman School of Public Health, Columbia University, New York, NY, United States

    Corresponding Author:

    Rebecca Schnall, MPH, PhD

    School of Nursing

    Columbia University

    560 West 168th Street

    New York, NY, 10032

    United States

    Phone: 1 2123426886

    Email: rb897@columbia.edu


    Background: Rigorous development of mobile technologies requires the use of validated instruments to evaluate the usability of these tools, which has become more relevant with the expansion of these technologies. Although various usability evaluation tools have been developed, there are relatively few simple evaluation instruments that have been validated across diseases and languages in mobile health (mHealth) information technology for use in multiple diseases.

    Objective: The purpose of this study is to validate the Korean version of the Health Information Technology Usability Evaluation Scale (Korean Health-ITUES) and assess its applicability for different health conditions.

    Methods: To develop the Korean Health-ITUES, we used a validation process involving the following 3 steps: (1) customization of the Health-ITUES for menstrual symptoms, (2) translation of the Health-ITUES from English into Korean, and (3) examination of the reliability and validity of the instrument. The translation process adhered to the World Health Organization (WHO) guidelines for translation and back-translation, expert review, and reconciliation.

    Results: The Korean Health-ITUES showed reliable internal consistency with Cronbach α=.951; meanwhile, factor loadings of the 20 items in the 4 subscales ranged from 0.416 to 0.892.

    Conclusions: The Health-ITUES demonstrated reliability and validity for its use in assessing mHealth apps’ usability in young Korean women with menstrual discomfort. Given the strong psychometric properties of this tool in Korean and English and across 2 different health conditions, the Health-ITUES is a valid and reliable instrument for assessing the usability of mHealth apps. The Health-ITUES is also a valid instrument for evaluating mHealth technologies, which are widely used by patients to self-manage their health and by providers to improve health care delivery.

    JMIR Med Inform 2022;10(1):e28621

    doi:10.2196/28621

    Keywords

    mobile application; menstruation; survey; questionnaire; translations; medical informatics; app; validity; reliability; usability; assessment; mHealth; evaluation 


    Background

    In the past decade, 1 of the most challenging components of technology development has been to ensure the usability of tools to ensure their quality in use [1,2]. To support use of the technology, usability must be assessed during the development process [3]. Usability is the measure of the quality of a user’s experience when interacting with a system—whether a website, mobile technology, or any user-operated device [4]. In other words, usability refers to how well users can navigate a system to achieve their goals and how satisfied they are with the process. A successful system needs to work for its users, and it needs to work well. However, many mHealth technologies have been made available to the public, with insufficient attention devoted to their design, development, and evaluation [5]. Technologies produced with poor design and inadequate consideration of the needs of their intended users will be difficult to learn, misused, or underutilized and will ultimately fail to accomplish their objectives [6]. For this reason, usability has been widely recognized as a critical consideration in evaluating the efficacy of technologies [7].

    Usability is especially critical for mobile technology, which is widely used in health care [8-10]. In fact, there were more than 800,000 mobile health (mHealth) apps in Apple App Store and Google Play Store in 2020. There is continued growth, with about 200 mHealth apps added each day, with some focusing on healthy eating, physical activity, and improved mental health [11-13]. To date, several studies have evaluated the effect of mHealth apps on disease management and prevention [14-17].

    Research to evaluate the usability of mHealth apps has been conducted for various health states, such as chronic obstructive pulmonary disease, HIV, obesity, depression, anxiety, dysmenorrhea, and premenstrual syndrome [18-26]. However, to ensure the rigor of mHealth technologies, it is necessary to create tools to evaluate the quality and usability of mHealth apps [27].

    The quality of mHealth apps requires understanding the context of their use and ensuring the apps’ usability [28]. Therefore, it is important to develop mHealth tools using rigorous usability evaluation tools. Although there are several mobile app assessment tools, most have a large number of items or have only been validated for a single disease or in a single language [29,30]. Given these limitations, there is a need for instruments that are validated across languages and diseases. This is especially true in South Korea, which has 1 of the highest rates (94%) of smartphone use in the world. This high penetration of smartphones has enabled the rapid integration of mHealth apps [31,32]. However, despite the high usage of mHealth apps in South Korea, there is a dearth of availability of simple instruments to assess their usability.

    The Study

    This study sought to translate the Health Information Technology Usability Evaluation Scale (Health-ITUES) from English into Korean and validate its use. The Health-ITUES is a customized questionnaire comprising 20 items and a modified version of the Health Information Technology Usability Evaluation Model (Health-ITUEM) [33]. The English version of the Health-ITUES has been previously validated through exploratory factor analysis (EFA) and confirmatory factor analysis after use by nurses [34] and community-dwelling adults with HIV [35]; however, it has not been translated into other languages and validated. This study translated the Health-ITUES into Korean and validated it in a sample of 244 women who experienced menstrual-related symptoms and used a menstrual tracking app called PINKDIARY.


    Sample/Participants

    This study was approved by the institutional review board of the Catholic University of Pusan (CUPIRB-2019-003) before the commencement of study activities. Inclusion criteria for this study were unmarried women >20 and <39 years of age who were previously or are currently using the menstrual-tracking app PINKDIARY for more than a month. The app records the highest usage rate in Korea, and as of October 2021, it was ranked sixth in the health and fitness category in Apple App Store. This app is the official app of the Korean Association of Obstetricians and Gynecologists. PINKDIARY is used to track menstruation and premenstrual symptoms. Features of the app include symptom records, doctor consultations, an online community, and a shopping mall for menstrual items (eg, pads, tampons, and menstrual cups). Marriage and age were also inclusion criteria, as these may have affected the participants’ usage of or experience with using the app.

    The sample size was set to ensure a minimum number of participants based on the number of items in the instrument. Nunnally [36] recommended a minimum participant ratio of 10 participants:1 survey item. In this study, the target sample size was between 200 and 250 after multiplying the number of questions (20) by 10. This estimate was based on an anticipated attrition rate of about 20%. In the past, the dropout rate in app-related studies was about 20%-50%. Since this study was not an intervention study, the dropout rate was estimated at 20% [37].

    Recruitment was conducted through the KakaoTalk (Kakao Corp) messenger and other online communities (eg, Everytime). Potential participants sent a screenshot of the PINKDIARY app to the researcher’s messenger to authenticate their use of the app. The online consent form and questionnaire were developed in Survey Monkey, and the link was sent to participants on the KakaoTalk messenger. After filling in the consent form and questionnaire, a 2000 won (about US $2) online coffee coupon was sent to the participants as a token of appreciation for their time.

    Step 1: Modification of the Health-ITUES

    We customized the Health-ITUES, which was previously validated in a sample of persons with HIV, for women with menstrual discomfort [25]. In this study, menstrual discomfort was defined as primary dysmenorrhea and premenstrual syndrome (PMS), which are the most common menstrual discomfort symptoms among women of reproductive age [38,39]. The modified version was reviewed by the senior author of this manuscript, RS.

    Step 2: Korean Health-ITUES

    The translation and back-translation followed WHO guidelines [40].

    Forward Translation

    To align with WHO guidelines, all items had to remain unchanged from their original meaning when translated and had to be translated into English by 2 or more translators. In this study, 2 Korea-born nurses who had lived in the United States for more than 5 years and received doctoral degrees in the United States translated the Health-ITUES from Korean into English. The translators then independently translated it from English into Korean. Following the independent translation, the 2 translators discussed the findings during 3 separate meetings until a consensus was reached.

    Expert Review

    The translated Health-ITUES was reviewed and reconfirmed by a bilingual (English/Korean) physician with dual training in medical informatics. The physician was fluent in both English and Korean. He checked whether the translated version applied to Korean-speaking people and revised the items with expressions or cultural differences that could cause different meanings to be conveyed. After the review process, the translation of the Health-ITUES from Korean into English was considered complete.

    Back-Translation

    Back-translation was conducted by a professional translator from an official translation company and a nurse who lived in the United States for over 10 years and was currently enrolled in a PhD program in the United States. The 2 translators translated the tool back into English and focused on culture and concepts rather than word differences, as was done during the translation process. Inconsistencies were reviewed by the first author to produce a back-translated version of the Health-ITUES.

    Expert Review

    The bilingual physician who participated in the first expert review after forward translation reviewed the back-translated version again. In this step, we focused on whether the Korean words before translation and the Korean words that were translated back had the same cultural and conceptual meanings, rather than focusing on whether they were completely identical.

    Original Author’s Review

    After completing the translation and back-translation, the senior author (RS) reviewed the English version of the Health-ITUES and confirmed the content.

    Step 3: Reliability and Validity

    Pilot Test

    According to WHO guidelines, the minimum number of pilot test respondents is 10 and should represent males and females of all ages and socioeconomic groups. However, in our study, due to time constraints and the fact that we reached data saturation after interviewing 5 women, we limited our sample size to only 5 respondents for this component part of the study [41]. Young women who majored in nursing and had previously used the menstrual app (for a minimum of 1 month to a maximum of 5 years) completed the survey items. Following completion of the survey, they provided feedback about the questionnaire’s items through an in-depth interview. During the interview, we asked the participants to justify their responses and whether they encountered any words in the Health-ITUES that were difficult to understand.

    Reliability and Validity Test

    Internal consistency reliability was measured using Cronbach’s alpha and Pearson’s correlation coefficient. Construct validity was analyzed using exploratory factor analysis. Exploratory factor analysis was used to confirm the predicted factor loadings based on the original instrument [42].

    Questionnaire

    The questionnaire comprised the items described in Table 1.

    Table 2 shows each item of the Health-ITUES and the Korean Health-ITUES customized for this study (see Multimedia Appendix 1 for the Korean Health-ITUES).

    Table 1. The questionnaire’s items.
    CategoriesItems (N=49), n (%)Item descriptionScale
    General characteristics5 (10)
    • Sex, marital status, age, residential area, job
      • ‎
    		N/Aa
    Smartphone experience and menstrual app usage11 (22)
    • Smartphone type, which features they used most often, how long, how often they used the menstrual app, etc
      • ‎
    		N/A
    Dysmenorrhea2 (4)
    • Pain on the first and second days of the menstrual period [23]
      • ‎
    • Visual analog scale
      • ‎
    PMSb11 (22)
    Subscale: emotion (4 items), water congestion (4 items), pain (2 items), and appetite (1 item)
    • The changes that participants experience before menstruation (an appetite item added to the shortened Premenstrual Assessment Form) [43]
      • ‎
    • 6-point Likert-type scale with responses ranging from 1 (not at all) to 6 (very severe change).
      • ‎
    • The higher the score, the more severe the symptoms.
      • ‎
    Korean Health-ITUESc20 (42)
    Subscale: impact (3 items), perceived usefulness (9 items), perceived ease of use (5 items), and user control (3 items)
    • Impact: high expectations for system impact and perceived usefulness as well as performance assessment of tasks through system usage
      • ‎
    • User-system interactions measured
      • ‎

    • 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree).
      • ‎
    • The higher the scale value, the higher the usability of the technology.
      • ‎
    • The overall Korean Health-ITUES score is the average of all items with the same weight for each item.
      • ‎

    aN/A: not applicable.

    bPMS: premenstrual syndrome.

    cHealth-ITUES: Health Information Technology Usability Evaluation Scale.

    Table 2. Health-ITUESa and Korean Health-ITUES.
    Health-ITUES [24]Korean Health-ITUES
    NumberItemNumberItem
    Impact

    		1I think Mobile Video Information Provider (mVIP)b would be a positive addition for persons with HIV.1I think PINKDIARYc would provide positive health outcomes for women with menstrual discomfort.

    		2I think mVIP would improve the quality of life of persons with HIV.2I think PINKDIARY would improve the quality of life of women with menstrual discomfort.

    		3mVIP is an important part of meeting my information needs related to symptom self-management.3PINKDIARY helps to meet the information needs for the self-management of my menstrual-related symptoms.
    Perceived usefulness

    		1Using mVIP makes it easier to self-manage my HIV-related symptoms.1Using PINKDIARY makes self-managing my menstrual-related symptoms easy.

    		2Using mVIP enables me to self-manage my HIV-related symptoms more quickly.2Using PINKDIARY allows me to manage my menstrual-related symptoms more quickly.

    		3Using mVIP makes it more likely that I can self-manage my HIV-related symptoms.3Using PINKDIARY makes self-managing my menstrual-related symptoms better.

    		4Using mVIP is useful for self-management for HIV-related symptoms.4Using PINKDIARY is useful for the self-management of my menstrual-related symptoms.

    		5I think mVIP presents a more equitable process for self-management of HIV-related symptoms.5I think PINKDIARY provides a more equitable process for the self-management of my menstrual-related symptoms.
    (Health equity refers to health equality, which means ensuring the optimal level of health for all people, regardless of income and educational level.)

    		6I am satisfied with mVIP for self-management of HIV-related symptoms.6I am satisfied with PINKDIARY for the self-management of my menstrual-related symptoms.

    		7I self-manage my HIV-related symptoms in a timely manner because of mVIP.7I can self-manage my menstrual-related symptoms in a timely manner thanks to PINKDIARY.

    		8Using mVIP increases my ability to self-manage my HIV-related symptoms.8Using PINKDIARY enhances my ability to self-manage my menstrual-related symptoms.

    		9I am able to self-manage my HIV-related symptoms whenever I use mVIP.9I can self-manage my menstrual-related symptoms when I use PINKDIARY.
    Perceived ease of use

    		1I am comfortable with my ability to use mVIP.1I am satisfied with my ability to use PINKDIARY.

    		2Learning to operate mVIP is easy for me.2It is easy for me to learn how to operate PINKDIARY.

    		3It is easy for me to become skillful at using mVIP.3It was easy for me to become skillful in using PINKDIARY.

    		4I find mVIP easy to use.4I find PINKDIARY easy to use.

    		5I can always remember how to log on to and use mVIP.5I always remember how to log on to and use PINKDIARY.
    User control

    		1mVIP gives error messages that clearly tell me how to fix problems.1PINKDIARY provides error messages that clearly explain how to solve problems with PINKDIARY.

    		2Whenever I make a mistake using mVIP, I easily and quickly recover.2I can recover quickly and easily whenever I make a mistake while using PINKDIARY.

    		3The information provided by mVIP (eg, online help, on-screen messages, and other documentations) is clear.3The information provided by PINKDIARY (eg, online help, screen messages, and other documents) is clear.

    aHealth-ITUES: Health Information Technology Usability Evaluation Scale.

    bmVIP: HIV self-management app.

    cPINKDIARY: menstrual app.

    Statistical Analysis

    Descriptive statistics were used to analyze the demographic characteristics of the study participants. Reliability and validity were analyzed using Cronbach α, Pearson correlation coefficient, and EFA. All analyses used IBM SPSS Statistics 24.0. Two-sided P values of <0.05 were considered statistically significant.

    Ethical Considerations

    All study participants provided informed consent, and the study design was approved by the appropriate ethics review board.


    Pilot Test

    Five women completed the draft version of the survey and provided the following feedback about the Health-ITUES:

    Overall, I think the questionnaire is easy to understand. However, in the case of items that need multiple responses, I would like for such questions to be clearly marked as “requiring multiple responses.”
    [Interviewee A]
    I like this online questionnaire because there are appropriate app icon figures with items. However, I need further explanation about Item 8: I cannot understand what ”equitable process” means.
    [Interviewee B]
    I did not have any problem with understanding the items in the questionnaire. However, I went to the next page without answering a few questions. If the respondent does not answer all the items, that is, if even one item has not been answered, please set the questionnaire such that it does not move to the next page.
    [Interviewee C]

    As a result of this feedback, item 8 was changed. More specifically, further explanation was added to clarify the term “equitable process” as follows: “Health equity refers to health equality, which means ensuring the optimal level of health for all people, regardless of income and educational level.”

    Demographic Characteristics

    A total of 244 unmarried female participants completed the Health-ITUES. The participants’ ages ranged from 20 to 36 years (mean=22.45, SD 3.039). Meanwhile, 127 (51.2%) participants majored in a health-related topic, and 117 (47.0%) participants majored in non-health-related topics for their university degrees. Participants resided in the following cities and provinces: Seoul (n=76, 30.5%), Gyeonggi-do (n=61, 24.6%), Chungcheong-do (n=52, 21.7%), Gyeongsang-do (n=48, 19.4%), and Else (7, 2.8%).

    Internal Consistency Reliability

    Internal consistency reliability was measured by Cronbach α, and the results are presented in Table 3. All items showed good Cronbach α values (>0.8) ranging from 0.83 to 0.94 (Table 3). All values were less than 0.95, which demonstrates that there was no redundancy among the items [44]. Internal scale correlation between items ranged from 0.45 to 0.71, indicating moderate to strong correlations. Notably, perceived usefulness was more highly correlated with impact than other subscales.

    Table 3. Descriptive statistics, internal scale consistency score, and internal scale correlation for Korean Health-ITUESa subscales (N=244).
    SubscaleMean±SDImpactPerceived usefulnessPerceived ease of useUser control


    		Cronbach αrCronbach αrCronbach αrCronbach αr
    Impact10.87±2.520.84—b——————
    Perceived usefulness34.51±6.55—0.707c0.94—————
    Perceived ease of use19.60±4.13—0.510c—0.660c0.91———
    User control10.16±2.43
    		0.446c
    		0.584c—0.647c0.83—

    aHealth-ITUES: Health Information Technology Usability Evaluation Scale.

    bNot applicable.

    cP<.05.

    Construct Validity

    Exploratory Factor Analysis

    EFA was performed to assess the construct validity of the Korean Health-ITUES items to extract potential factors. Results are reported in Table 4. The Kaiser-Meyer-Olkin measure of sampling adequacy (MSA) was 0.942, indicating that the data in this study were suitable for factor analysis. In addition, as a result of the Bartlett sphericity test, the correlation between the Korean Health-ITUES variables was recognized based on the significance level of .05 with x2=3929.635 and P<.01. Thus, 4 subfactors were extracted.

    Table 4. Principal axis factoring with varimax rotation.
    ItemCommonalityComponent

    		Perceived usefulnessPerceived ease of useImpactUser control
    usefulness80.8050.831a0.2080.1490.223
    usefulness90.7640.8110.1970.2090.154
    usefulness70.6770.7470.2120.1140.249
    usefulness30.6940.7350.2220.3060.105
    usefulness40.6590.7220.1430.3160.137
    usefulness20.6630.7000.1750.3180.201
    usefulness10.7280.6870.2340.4010.200
    usefulness60.7010.6780.3230.3140.196
    easeofuse10.7150.5740.5370.1830.251
    usefulness50.5390.4160.1620.4100.414
    easeofuse20.8880.2200.8920.1610.136
    easeofuse30.9010.2510.8630.1970.233
    easeofuse40.9040.3140.8560.1710.209
    easeofuse50.5970.1770.5790.0800.474
    impact10.7890.3030.1240.7980.065
    impact20.7480.3090.1190.7980.205
    impact30.7080.3740.2840.6870.126
    control10.8050.2040.1150.0960.861
    control20.7650.2140.4290.0620.729
    control30.6920.2660.2550.2880.688

    aItalics indicate the number of items corresponding to the component.


    Principal Findings

    A major challenge to technology development is ensuring the usability of the tools [1-3]. However, many mHealth tools are currently available with little attention to their usability [5]. Technologies produced with poor design and inadequate consideration of the needs of their intended users are often difficult to use, and the consumers often cannot accomplish their goals, as a result [6]. Therefore, usability has been widely recognized as an important factor in the development of technology [7].

    To address the need for understanding the usability of technology, various studies have been conducted [29,30]. However, each assessment tool could not be easily used, because it had too many questions, took a long time to answer due to difficult questions, was developed for specific users, or was developed only in English [23,29,30,45]. Therefore, the Health-ITUES, a simple and verified tool for multiple populations, was chosen for translation and validation. Because the Health-ITUEM, which is the basis of the Health-ITUES, was developed based on several usability models with strong reliability and validity, including the technology acceptance model (TAM) and ISO9241-11 [46-48]. Additionally, the Health-ITUES was validated in a sample of nurses and community-dwelling adults with HIV [33-35].

    This study sought to not only translate but also verify the Health-ITUES considering the Korean context. In step 1, the authors modified the Health-ITUES that was used to evaluate an app for adults with HIV to an app for women suffering from menstrual discomfort. In Korea, there are fewer HIV-infected people compared to the United States. Additionally, because of negative views regarding homosexual contact, HIV-infected people are more likely to not disclose themselves [49]. As of October 2021, when searching for HIV/AIDS-related apps in Apple App Store and Google Play Store, it was difficult to find mHealth for patients with HIV in Korean. To obtain sufficient app users for the validation of the translated tool, the Health-ITUES was modified to enable usability evaluation of the menstrual app PINKDIARY for managing dysmenorrhea experienced by 75% of domestic women [38,50].

    In step 2, the Health-ITUES was translated based on WHO guidelines [40]. These guidelines have been used to translate various instruments into many languages, such as Japanese and Arabic, and are not limited to Korean [51-53]. The process of translation and adaptation of instruments were as follows: forward translation, back-translation, and cognitive testing. In this study, 3 experts who majored in nursing or medical informatics and 1 professional English/Korean translator participated in the forward translation and back-translation, respectively. The 2 translators independently translated and reconciled any discrepancies after the forward translation following WHO guidelines. This study was further strengthened by the review of the Korean Health-ITUES by 1 of the authors, who validated the Health-ITUES for mHealth technology. This process ensured that the original meanings of the Health-ITUES items were retained.

    Another strength of this reliability and validity is our study sample size. The sample size exceeded the number of items, 20 × 10, and the participants live in various regions in Korea. Their field of work (major) is also not biased, so it can be said that geographical biases are small.

    Moreover, in step 3, internal scale correlation ranged from 0.45 to 0.71, which indicates moderate to strong correlations. Notably, perceived usefulness was more highly correlated with impact than the other factors. The results mirrored the findings from the validation study using the Health-ITUES in a sample of adults with HIV [34]. In EFA, in the case of values for usefulness, 5 items (“I think PINKDIARY provides a more equitable process for the self-management of my menstrual-related symptoms.”) were included as 3 components because the values were so similar: 0.416, 0.410, and 0.414. As a result, we decided to keep this item with the first component perceived usefulness with which it is most closely conceptually aligned [54].

    Limitations

    One limitation of this study is that when recruiting survey participants, only the experience of using the menstruation-related mHealth app was checked and no restrictions were placed on the past period of use. Future research should recruit participants by suggesting clear past usage periods based on objective evidence. However, this study successfully translated the Health-ITUES from English into Korean and validated it, and this instrument can be used to evaluate the usability of mHealth apps. These findings will contribute to the systematic evaluation of the rapidly growing field of mHealth apps.

    Conclusion

    The Health-ITUES demonstrated reliability and validity for use in assessing mHealth apps’ usability in young Korean women with menstrual discomfort. Given the strong psychometric properties of this tool in Korean and English and across 2 different health conditions, the Health-ITUES is a strong instrument for evaluating the usability of mHealth apps.

    Acknowledgments

    This research was supported by the Basic Science Research Program through the National Research Foundation of Korea, funded by the Ministry of Education (NRF-2018R1C1B5030802, NRF-2021R1C1C1004505). RS was supported by the National Institute of Nursing Research of the National Institutes of Health (award no. K24NR018621). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

    Authors' Contributions

    All authors had full access to all data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors were involved in the study concept and design. JL performed data acquisition and statistical analysis of the data, drafted the manuscript, and provided administrative, technical, and material support. RS performed data interpretation, critical revision of the manuscript for important intellectual content, and study supervision and gave the final approval of the version to be submitted. JL and RS obtained funding.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    The Korean version of the Health-ITUES. Health-ITUES: Health Information Technology Usability Evaluation Scale.

    PNG File , 70 KB
    1. Folmer E, Bosch J. Architecting for usability: a survey. J Syst Softw 2004 Feb;70(1-2):61-78. [CrossRef]
    2. Bengtsson PO, Lassing NH, Bosch J, van Vliet H. Analyzing Software Architectures for Modifiability. Ronneby, Sweden; 2000.   URL: https://www.diva-portal.org/smash/get/diva2:838104/FULLTEXT01.pdf [accessed 2021-03-08]
    3. Holzinger A. Usability engineering methods for software developers. Commun ACM 2005 Jan;48(1):71-74. [CrossRef]
    4. Your Guide for Developing Usable and Useful Web Sites.   URL: http://www.usability.gov/ [accessed 2022-01-03]
    5. Nilsen W, Kumar S, Shar A, Varoquiers C, Wiley T, Riley WT, et al. Advancing the science of mHealth. J Health Commun 2012 May 02;17 Suppl 1(sup1):5-10. [CrossRef] [Medline]
    6. Maguire M. Methods to support human-centred design. Int J Hum-Comput Stud 2001 Oct;55(4):587-634. [CrossRef]
    7. Shackel B. Usability: context, framework, definition, design and evaluation. Interact Comput 2009 Dec;21(5-6):339-346. [CrossRef]
    8. Agarwal S, LeFevre AE, Lee J, L'Engle K, Mehl G, Sinha C, WHO mHealth Technical Evidence Review Group. Guidelines for reporting of health interventions using mobile phones: mobile health (mHealth) evidence reporting and assessment (mERA) checklist. BMJ 2016 Mar 17;352:i1174. [CrossRef] [Medline]
    9. Hood M, Wilson R, Corsica J, Bradley L, Chirinos D, Vivo A. What do we know about mobile applications for diabetes self-management? A review of reviews. J Behav Med 2016 Dec 13;39(6):981-994. [CrossRef] [Medline]
    10. Huzooree G, Kumar Khedo K, Joonas N. Pervasive mobile healthcare systems for chronic disease monitoring. Health Inform J 2019 Jun 02;25(2):267-291 [FREE Full text] [CrossRef] [Medline]
    11. Clement J. Number of mHealth Apps Available at Google Play from 1st Quarter 2015 to 2nd Quarter 2020.   URL: https://www.statista.com/statistics/779919/health-apps-available-google-play-worldwide/ [accessed 2021-12-28]
    12. Aitken B. The Growing Value of Digital Health.   URL: https://www.iqvia.com/insights/the-iqvia-institute/reports/the-growing-value-of-digital-health [accessed 2022-01-03]
    13. McKay FH, Wright A, Shill J, Stephens H, Uccellini M. Using health and well-being apps for behavior change: a systematic search and rating of apps. JMIR Mhealth Uhealth 2019 Jul 04;7(7):e11926 [FREE Full text] [CrossRef] [Medline]
    14. Kuhns LM, Garofalo R, Hidalgo M, Hirshfield S, Pearson C, Bruce J, et al. A randomized controlled efficacy trial of an mHealth HIV prevention intervention for sexual minority young men: MyPEEPS mobile study protocol. BMC Public Health 2020 Jan 15;20(1):65 [FREE Full text] [CrossRef] [Medline]
    15. van der Merwe MN, Mosca R, Swanepoel DW, Glascoe FP, van der Linde J. Early detection of developmental delays in vulnerable children by community care workers using an mHealth tool. Early Child Dev Care 2018 Jun 04;189(5):855-866. [CrossRef]
    16. Palmer M, Barnard S, Perel P, Free C. Mobile phone-based interventions for improving adherence to medication prescribed for the primary prevention of cardiovascular disease in adults. Cochrane Database Syst Rev 2018 Jun 22;6:CD012675 [FREE Full text] [CrossRef] [Medline]
    17. Ravn Jakobsen P, Hermann AP, Søndergaard J, Wiil UK, Clemensen J. Help at hand: women's experiences of using a mobile health application upon diagnosis of asymptomatic osteoporosis. SAGE Open Med 2018 Oct 29;6:2050312118807617 [FREE Full text] [CrossRef] [Medline]
    18. Beauchemin M, Gradilla M, Baik D, Cho H, Schnall R. A multi-step usability evaluation of a self-management app to support medication adherence in persons living with HIV. Int J Med Inform 2019 Feb;122:37-44 [FREE Full text] [CrossRef] [Medline]
    19. Boer L, Bischoff E, van der Heijden M, Lucas P, Akkermans R, Vercoulen J, et al. A smart mobile health tool versus a paper action plan to support self-management of chronic obstructive pulmonary disease exacerbations: randomized controlled trial. JMIR Mhealth Uhealth 2019 Oct 09;7(10):e14408 [FREE Full text] [CrossRef] [Medline]
    20. Cho H, Powell D, Pichon A, Kuhns LM, Garofalo R, Schnall R. Eye-tracking retrospective think-aloud as a novel approach for a usability evaluation. Int J Med Inform 2019 Sep;129:366-373 [FREE Full text] [CrossRef] [Medline]
    21. Grossman JT, Frumkin MR, Rodebaugh TL, Lenze EJ. mHealth assessment and intervention of depression and anxiety in older adults. Harv Rev Psychiatry 2020 Apr 20;28(3):203-214. [CrossRef]
    22. Lee J, Kim J. Development and efficacy testing of a social network-based competitive application for weight loss. Telemed J E Health 2016 May;22(5):410-418. [CrossRef] [Medline]
    23. Lee J, Kim J. Can menstrual health apps selected based on users' needs change health-related factors? A double-blind randomized controlled trial. J Am Med Inform Assoc 2019 Jul 01;26(7):655-666 [FREE Full text] [CrossRef] [Medline]
    24. Levy J. Of mobiles and menses: researching period tracking apps and issues of response-ability. Stud Home Comm Sci 2018 Jan 08;11(2):108-115. [CrossRef]
    25. Moglia ML. Evaluation of smartphone menstrual cycle tracking applications using an adapted applications scoring system. Obstet Gynecol 2016;127(6):1153-1160. [CrossRef]
    26. Pray W. Nonprescription Product Therapeutics. Philadelphia, PA: Lippincott Williams & Wilkins; 2006.
    27. Kumar S, Nilsen WJ, Abernethy A, Atienza A, Patrick K, Pavel M, et al. Mobile health technology evaluation: the mHealth evidence workshop. Am J Prev Med 2013 Aug;45(2):228-236 [FREE Full text] [CrossRef] [Medline]
    28. Ben-Zeev D, Kaiser SM, Brenner CJ, Begale M, Duffecy J, Mohr DC. Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia. Psychiatr Rehabil J 2013 Dec;36(4):289-296 [FREE Full text] [CrossRef] [Medline]
    29. Faieta J, Hand BN, Schmeler M, Onate J, Digiovine C. Health app review tool: matching mobile apps to Alzheimer's populations (HART Match). J Rehabil Assist Technol Eng 2020 Sep 23;7:2055668320938604 [FREE Full text] [CrossRef] [Medline]
    30. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and validation of the user version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016 Jun 10;4(2):e72 [FREE Full text] [CrossRef] [Medline]
    31. Intralink. Digital Health South Korea Market Intelligence Report. 2019 Jun 01.   URL: https://www.intralinkgroup.com/getmedia/3153c79b-463d-47c7-84e6-56848c98aab7/Intralink-ReportLife [accessed 2022-01-03]
    32. Silver L. Smartphone Ownership is Growing Rapidly Around the World, but Not Always Equally. 2019 Feb.   URL: https://tinyurl.com/ywpub5p4 [accessed 2022-01-03]
    33. Brown W, Yen P, Rojas M, Schnall R. Assessment of the Health IT Usability Evaluation Model (Health-ITUEM) for evaluating mobile health (mHealth) technology. J Biomed Inform 2013 Dec;46(6):1080-1087 [FREE Full text] [CrossRef] [Medline]
    34. Schnall R, Cho H, Liu J. Health Information Technology Usability Evaluation Scale (Health-ITUES) for usability assessment of mobile health technology: validation study. JMIR Mhealth Uhealth 2018 Jan 05;6(1):e4 [FREE Full text] [CrossRef] [Medline]
    35. Yen P, Sousa KH, Bakken S. Examining construct and predictive validity of the Health-IT Usability Evaluation Scale: confirmatory factor analysis and structural equation modeling results. J Am Med Inform Assoc 2014 Oct 01;21(e2):e241-e248 [FREE Full text] [CrossRef] [Medline]
    36. Nunnally J. Psychometric Theory, 2nd edition. New York, NY: McGraw-Hill; 1978.
    37. Torous J, Lipschitz J, Ng M, Firth J. Dropout rates in clinical trials of smartphone apps for depressive symptoms: a systematic review and meta-analysis. J Affect Disord 2020 Feb 15;263:413-419. [CrossRef] [Medline]
    38. Quick F, Mohammad-Alizadeh-Charandabi S, Mirghafourvand M. Primary dysmenorrhea with and without premenstrual syndrome: variation in quality of life over menstrual phases. Qual Life Res 2019 Jan 12;28(1):99-107. [CrossRef] [Medline]
    39. Bilir E, Yıldız Ş, Yakın K, Ata B. The impact of dysmenorrhea and premenstrual syndrome on academic performance of college students, and their willingness to seek help. Turk J Obstet Gynecol 2020 Sep 2;17(3):196-201 [FREE Full text] [CrossRef] [Medline]
    40. World Health Organization. Process of Translation and Adaptation of Instruments.   URL: https://www.who.int/substance_abuse/research_tools/translation/en/ [accessed 2021-12-28]
    41. Faulkner S, Trotter S. Data saturation. In: The International Encyclopedia of Communication Research Methods. Hoboken, NJ: John Wiley & Sons; 2017.
    42. Plichta Kellar S, Kelvin E. Munro's Statistical Methods for Health Care Research, 6th Edition. Philadelphia, PA: Lippincott Williams & Wilkins; 2005:1.
    43. Allen SS, McBride CM, Pirie PL. The shortened premenstrual assessment form. J Reprod Med 1991 Nov;36(11):769-772. [Medline]
    44. Tavakol M, Dennick R. Making sense of Cronbach's alpha. Int J Med Educ 2011 Jun 27;2:53-55 [FREE Full text] [CrossRef] [Medline]
    45. Lee J, Kim J. Method of app selection for healthcare providers based on consumer needs. Comput Inform Nurs 2018;36(1):45-54. [CrossRef]
    46. Davis FD, Bagozzi RP, Warshaw PR. User acceptance of computer technology: a comparison of two theoretical models. Manag Sci 1989 Aug;35(8):982-1003. [CrossRef]
    47. ISO. Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs).   URL: https://www.iso.org/standard/16883.html [accessed 2022-01-03]
    48. Yen P, Bakken S. Review of health information technology usability study methodologies. J Am Med Inform Assoc 2012;19(3):413-422 [FREE Full text] [CrossRef] [Medline]
    49. Kim JM, Kim NJ, Choi JY, Chin BS. History of acquired immune deficiency syndrome in Korea. Infect Chemother 2020 Jun;52(2):234-244 [FREE Full text] [CrossRef] [Medline]
    50. Lee EH, Kim JI, Kim HW, Lee HK, Lee SH, Kang NM, et al. Dysmenorrhea and menstrual attitudes in adult women. Korean J Women Health Nurs 2003;9(2):105. [CrossRef]
    51. Cho E, Choi M, Kim E, Yoo IY, Lee N. Construct validity and reliability of the Korean version of the practice environment scale of nursing work index for Korean nurses. J Korean Acad Nurs 2011 Jun;41(3):325-332. [CrossRef] [Medline]
    52. Toma G, Guetterman TC, Yaqub T, Talaat N, Fetters MD. A systematic approach for accurate translation of instruments: experience with translating the Connor–Davidson Resilience Scale into Arabic. Meth Innov 2017 Dec 21;10(3):205979911774140. [CrossRef]
    53. Kurosawa S, Matsushima M, Fujinuma Y, Hayashi D, Noro I, Kanaya T, et al. Two principal components, coping and independence, comprise patient enablement in Japan: cross sectional study in Tohoku area. Tohoku J Exp Med 2012 Jun;227(2):97-104 [FREE Full text] [CrossRef] [Medline]
    54. Hair JF, Black WC, Babin B, Anderson RE. Multivariate Data Analysis: A Global Perspective, 7th Edition. Upper Saddle River, NJ: Prentice Hall; 2009.

    ‎
    EFA: exploratory factor analysis
    Health-ITUEM: Health Information Technology Usability Evaluation Model
    Health-ITUES: Health Information Technology Usability Evaluation Scale
    mHealth: mobile health
    MSA: measure of sampling adequacy
    PMS: premenstrual syndrome
    TAM: technology acceptance model
    WHO: World Health Organization

    Edited by G Eysenbach; submitted 08.03.21; peer-reviewed by D Yoon, J Mitchell, H Ranjani; comments to author 30.03.21; revised version received 28.04.21; accepted 30.11.21; published 24.01.22

    Copyright

    ©Jisan Lee, Rebecca Schnall. Originally published in JMIR Medical Informatics (https://medinform.jmir.org), 24.01.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.