The National Center for Education Statistics defines competency as a combination of skills, abilities, and knowledge needed to perform a specific task [1]. The 21st century health informatics jobs will require specific skills such as collecting data from wireless medical devices and integrating real-time data analytics and artificial intelligence (AI) algorithms in clinical patient monitoring apps. Even though health informatics is a distinct interdisciplinary field that provides various paths to different careers and covers a variety of topics, the specific skill sets required by different employers vary owing to the increasing rate of technological developments [2]. In addition, the skills needed for health informaticians vary significantly depending on the position [3], and health informatics students need skills pertinent to their professional experience for their future career paths [4]. However, there are still significant gaps in workforce skills training, and studies examining students’ perspectives on required skill sets are limited. As academicians, students, employers, and people working in the health care industry have different perspectives and priorities for required informatics skills, identifying health informatics skill sets for graduate students has always been a challenge. Although students with a clinical background might need mobile health (mHealth) skills to diagnose and treat patients, those with information technology background might need advanced technical and programming skills to design and develop patient-centered health information systems, connected medical devices, consumer-directed mHealth apps, the internet of things–linked wearable solutions and analytics solutions that utilize machine learning, and personalized medicine apps that use AI algorithms [5].

The first international recommendations to develop health informatics educational activities were published by the International Medical Informatics Association (IMIA) in 2000 [6]. IMIA determined 4 knowledge or skill domains for international information technology users and biomedical and health informatics specialists: (1) methodology and technology for the processing of data, information, and knowledge in medicine and health care; (2) medicine, health and bioscience, and health system organization; (3) informatics and computer science, mathematics, and biometry; and (4) optional modules (Table 1). IMIA has been developing self-assessment requirements, pilot-testing the procedure, and conducting site visits since 2012. The organization also conducted a strengths, weaknesses, opportunities, and threats analysis for their accreditation process and determined that their educational recommendations could be used on all continents. Currently, IMIA is the only organization that develops international accreditation competencies [7-9].

With the adoption of digital technologies around the world, several countries have focused on education initiatives to improve populations’ 21st century digital skills [10]. Professional organizations have developed their country-specific health informatics competencies; for instance, the Health Informatics Society of Australia developed a competency framework for the Australian health care system [11].

As there are several international and national initiatives to develop health informatics competencies, determining the standard comprehensive health information skill sets has always been a challenging task because of the continually evolving technology. National health informatics organizations and accreditation agencies have been specifying their own standards because of the lack of universal standards. For example, Canada’s Health Informatics Association (Digital Health Canada, known as COACH before 2017) developed 51 competencies about information management, information technology, clinical/health services, Canadian Health System, organizational and behavioral management, project management, and analysis and evaluation in 2009 [12]. The Australian Health Informatics Education Council identified 45 core competencies for the Australian workforce [13]. In the United States, the Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM) and the American Medical Informatics Association (AMIA) have been establishing curriculum requirements. Even though CAHIIM’s 2012 curriculum requirements did not include any skill sets training, the revised 2017 recommendations were more specific and included skill definition for 7 foundational domains (Table 1) [14].

Another notable development was the formation of the eHealth Collaboration Workforce Development Workgroup in 2013. A Web-based database about health information technology competencies to identify the gaps between information and communication technology (ICT) competency and knowledge deficiencies was built by the United States and the European Union (EU) [15]. This comprehensive database encompasses 5 domains (administration, direct patient care, engineering/information systems/ICT, and informatics and research/biomedicine) and consists of 33 competency areas including electronic health (eHealth); mHealth; telehealth; data compiling, analysis, modeling, and reporting; and clinical decision support and pathways. This project was funded by Horizon 2020, which was the EU’s most significant research program [16]. Furthermore, the EU-US eHealth Work Project, which began in September 2016, currently conducts research to map skills and competencies and develop tools. This initiative plans to publish a comprehensive set of foundational curricula and advance eHealth/health information technology workforce when they complete the project [14,15].

The accreditation process evaluates an academic institution’s effectiveness in achieving its stated mission, and the graduate education programs that participate in the voluntary accreditation process should comply with the regional, national, or independent accrediting agencies’ core curriculum requirements. This is an important process to ensure the accountability of academic training programs and is widely considered as the de facto standard for quality assessment and continuous improvement.

In the United States, although the CAHIIM accredits undergraduate and graduate health informatics programs, the American Health Information Management Association (AHIMA) and the Commission on Certification for Health Informatics and Information Management certify individuals. In 2012, CAHIIM published the curriculum requirements for a master’s in health informatics degree and 3 mandatory facets about (1) the design, analysis, implementation, and management of information systems; (2) sociotechnical aspects; human-computer interaction; and structure, function, and transfer of information; and (3) computer networks, security, programming, database, and systems administration were determined [14]. In addition, one optional facet about optional courses such as medical terminology; anatomy; physiology; quantitative, qualitative, and mixed methods; and epidemiology was recommended (Table 1) [14].

In 2017, CAHIIM revised the accreditation standards for master’s degree programs in health informatics and published a revised version of core competencies that consists of the following foundational domains: (1) health; (2) information science and technology; (3) social and behavioral science; (4) health information science and technology; (5) human factors and sociotechnical systems; (6) social and behavioral aspects of health; (7) social, behavioral, and information science and technology applied to health; (8) professionalism; (9) interprofessional collaborative practice; and (10) leadership (Table 1) [14,17,18].

After the discussion about the need to explore the description of core informatics competencies in 2001, the AMIA education committee proposed establishing a medical informatics certification program the following year. AMIA’s working groups have been working on the definition and description of clinical informatics subspecialty and determining core competencies for biomedical and health informatics [19], whereas the Centers for Disease Control and Prevention has been leading a similar initiative for public health informaticians [20]. AMIA joined CAHIIM in 2015 and acknowledged the need for competency descriptions in a usable form. The Health Informatics Accreditation Council also started working on the revision of CAHIIM’s Health Informatics Accreditation Standard [21]. AMIA published the core competencies for health informatics education as an organizational member. Table 2 lists the foundational domains that list skills in CAHIIM’s revised skill recommendations document [22]. This new skills framework consists of various competency titles related to the leadership; professionalism; interprofessional collaborative practice; social, behavioral, and information science; social and behavioral aspects of health; human factors; and health information science and technology foundational domains, but it does not provide specific details.

New job opportunities for health informatics professionals require specific skill sets to utilize new cutting-edge, patient-focused delivery tools. Eligibility requirements for an advanced health informatics certification were proposed in 2016 [23]. Following this proposal, AMIA conducted the first informatics workforce survey in 2017 to build an inventory of informaticians’ unique skills and knowledge in the United States. The workforce survey evaluated professionals’ and students’ opinions on pursuing professional credentials and essential tasks in their informatics work [24]. Similarly, Digital Health Canada has conducted several competency surveys in Canada [25].

Academicians have also been discussing the integration of skills training into the health informatics curriculum for a long time. For example, new educational approaches related to emerging health information technologies were described, efforts to increase electronic health record (EHR) adoption were discussed, and hands-on exposure to health information systems during the graduate education was recommended to provide the necessary skills to solve interoperability issues [26]. Although regional, national, and independent accrediting agencies determine the core curriculum requirements for health informatics educational programs, these standards are not prescriptive. Health informatics faculty members who work in academic institutions, health informatics departments, and programs are expected to follow up on the changing requirements and update the content of their curriculum continuously. In addition, the Health Information Technology Workforce curriculum includes hands-on laboratory courses and encourages adding internship opportunities in the curriculum [27].

In addition, the IMIA’s working group encouraged the international health informatics community to begin a discussion on various big data and data training skills [28]. IMIA determined 3 domains and 12 learning outcomes that are related to data training and skills. These learning outcomes focus on health data management principles; structure and design principles of health records; principles of data representation and analysis; ethical and security issues; nomenclatures, vocabularies, terminologies, ontologies, and taxonomies; health administration and economics; basic informatics terminology; ability to communicate electronically; and methods of practical and theoretical informatics, mathematics, biometry, and epidemiology [29]. Although the digital divide is still a challenge, mobile broadband networks have reached 84% of the global population, and 46% of households have internet access around the world [30].

A number of health informatics students acquire skills training on the job rather than during their formal education, and recent studies emphasize the need for new models for skills acquisition [4,5]. However, the research on technology skills training in graduate health informatics curricula is still insufficient. The Office of the National Coordinator (ONC) for health informatics technology program recommended the integration of hands-on experience into the curriculum [27], but relatively few programs formally integrated digital technology skills training into their curriculum, and core technical skills to use digital technologies for medical apps were not well articulated in graduate health informatics and medical education programs [4].

According to the American Society for Training and Development, skills gaps in the organizations have been growing [31]. An EHR software called the Veterans Information Systems and Technology Architecture is the only hands-on training recommendation of the Workforce Development Program [27]. Although some nursing informatics programs have been integrating experiential learning in their graduate programs [32], most nursing schools provide limited technology training to teach how to enter, manage, and use data using various types of EHRs in traditional ways [33]. Similarly, most medical education programs limit technology-related training with the effective use of EHRs [34]. Conversely, an AMIA and AHIMA joint task force developed a detailed EHR core competencies matrix tool for different disciplines. This was one of the most important initiatives related to the development of EHR utilization skills in the clinical settings and was followed by similar initiatives [35].

Moreover, health informatics students need additional competencies to design and develop patient-centered health information systems, mine and analyze health care data, and use telemedicine and wireless remote monitoring systems. Evolving information technology and the growing number of medical devices and software apps for mobile devices require qualified workers with new skill sets, which were not included in the health informatics curriculum in the past. Overall, 5 employer-desired skill categories in bioinformatics—general, computational, biology, statistics and mathematics, and bioinformatics—were determined [36]. A recent report also emphasized health care organizations’ needs for analytics technology skills [37].

One of the major competency-based training initiatives was the Technology Informatics Guiding Education Reform (TIGER), which was established in 2006 to review informatics competencies for nursing students and practicing nurses. This initiative identified knowledge and skill set needs, which subsequently led to the development of an informatics competency framework for nurses that consists of basic computer skills, information literacy, and information management components. The TIGER Informatics Competencies Collaborative published their final report in 2009, and complex demands in health care led to the development of other national collaborative projects [38]. In addition, the Quality and Safety Education for Nurses Institute developed 6 competencies to provide safe and effective care, and one of them was focused explicitly on informatics skills to support clinical decision support and knowledge management care [39].

Despite several recommendations by professional organizations, a skills training framework for health informatics students is still not clearly defined. Existing skills training recommendations mostly focus on EHR training, and they generally do not include mHealth, home care, remote monitoring, AI, and data science training skills [4].

A study to determine students’ confidence in specific health informatics skills was conducted. For this purpose, published health informatics competencies were evaluated by two researchers independently, and a questionnaire to investigate skill sets of graduate Master of Science (MS) in Health Informatics students was developed by surveying core facility directors [36], IMIA [29] and CAHIIM’s curriculum requirements [16,17], ONC for Health Information Technology Workforce Development Program’s recommendations [40], TIGER initiative’s final report [38], the Association of American Medical College report [31], and the Health Informatics Society of Australia’s health informatics skill recommendations [11]. To measure students’ specific software skills, the most widely used statistics and office app packages were selected.

The questionnaire was divided into three parts. Part 1 consisted of demographic questions. Part 2 collected information about self-assessed skill sets using Likert scale questions, and 24 health informatics skills were determined for the second part of the questionnaire. Part 3 explored students’ suggestions for a new curriculum using open-ended questions.

A Web-based questionnaire was sent to a total of 223 enrolled and graduated students in the master’s degree program. Overall, 45.7% (102/223) of the participants completed the questionnaire within 2 months of the survey period, and all survey submissions were suitable for analysis. Table 3 illustrates the general demographic characteristics of the participants.

The questionnaire was tested on a small sample of respondents to identify problems with the construction and potential problems with the unclear wording. Face validity was established by an expert faculty member. The questionnaire was assessed, and the feedback about the clarity, friendliness of questions, and consistency was provided. Cronbach alpha was used to assess internal consistency, and it ranged from .9947 to .9952 (N=102). The overall reliability demonstrated excellent internal consistency.

The inclusion criteria included the participants’ informed consent and being enrolled in or graduated from the MS in health informatics program at Adelphi University in Garden City, New York, United States. As skills training is not included in the curriculum, current students do not receive formal hands-on training. Therefore, all enrolled and graduated students were included in the study, and survey results were not divided.

An institutional review board–approved questionnaire was distributed to all graduated and enrolled students. The participants received the consent form and instructions to complete a Web-based questionnaire, and 4 reminder emails were sent at 1-week intervals. The survey was anonymous. Participation in the study was voluntary, and there was no grade or compensation.

Among the respondents, 30.4% (31/102) were males, and 70.0% (71/102) were females. The largest percentage of respondents was aged less than 34 years; nearly one-third (31/102, 30.4%) of the participants were aged 35 to 54 years, and only 2 participants were older than 55 years. The majority of the respondents were currently enrolled in the program (73/102, 71.6%). Most of the respondents had an information technology– or health care–related occupation (74/102, 72.5%), and only 10.8% (11/102) of participants were not employed (Table 3).

Benner’s 5-level model of skill acquisition framework (novice, advanced beginner, competent, proficient, and expert) was applied to assess students’ level of confidence [41]. Descriptive statistics were used to describe the students’ self-assessments of important skills in the forms of mean, standard deviation, and frequency. As health informatics accreditation competencies do not contain specific skill training recommendations and these components are not included in the current curriculum, graduate and enrolled students’ responses were analyzed together. There were 24 items, and the margin of error was determined as 7.16, assuming a 95% confidence level.

Respondents initially rated themselves higher on Microsoft Word essential skills. For skills to insert a table of contents, footnotes, endnotes, and cross-references, 84.3% (86/102) of respondents rated themselves as extremely confident or very confident, 9.8% (10/102) as moderately confident, and 6.9% (7/102) as not at all confident and slightly confident. The mean was 4.21 (expert).

Participants rated themselves as proficient in Skills in evaluating health information systems (mean 3.14), Skills in training staff on system use, troubleshooting software and hardware issues (mean 3.28), Skills in performing math using Microsoft Excel and enter a calculation formula (mean 3.81), Skills in choosing evidence-based resources (mean 3.75), and Skills in compiling data from secondary sources (mean 3.20). For advanced Microsoft Excel skills such as calculating sample variance and standard deviation, 45.1% (46/102) of participants rated themselves as extremely confident or very confident, 35.3% (36/102) as moderately confident, and 19.6% (20/102) as not at all confident and slightly confident (mean 3.48; Table 4).

Respondents rated themselves as competent in Skills in programming mobile health informatics apps (mean 2.24), Skills in designing and leading health informatics projects (mean 2.84), Skills in setting up new businesses (mean 2.61), Skills in mining and analyzing data (mean 3.00), Skills in interpreting inferential statistics (mean 2.63), Skills in developing data visualization techniques (mean 2.42), Skills in using PICO to plan a search (mean 2.50), Skills in developing a database using Microsoft Access (mean 2.82), Skills in assessing data integrity and assessing data reliability (mean 2.94), Skills in evaluating the use of data capture technologies (mean 2.82), Skills in designing databases (mean 2.34), Skills in evaluating privacy and security infrastructure (mean 2.34), Skills in using Microsoft Word’s macro commands, creating dialog boxes, and understanding the notions of Visual Basic Application programming (mean 2.62), Skills in developing machine learning applications (mean 2.30), Skills in developing software to collect, organize, analyze, and interface with data (mean 2.35), and Skills in performing statistical tests using SPSS (mean 2.74; Table 4).

For AI app development skills, 6.9% (7/102) of the participants rated themselves as extremely confident or very confident, 14.7% (15/102) as moderately confident, 79.4% (81/102) as not at all confident or slightly confident. The mean was 1.80 (advanced beginner; Table 4).

The qualitative data analysis process to identify patterns and themes was inspired by Braun and Clark’s thematic analysis method [42]. The 6 steps of thematic analysis were used, and 4 themes emerged from the data:

Even though the demand for health informatics graduates has been changing, to the best of our knowledge, the number of studies that focus on hands-on health informatics skills training is limited. Recently, the Institute of Education Sciences developed a classification system called the Integrated Postsecondary Education Data System to track and report the fields of study [43]. However, this classification system does not cover all potential career paths as the professions related to health informatics fall under several occupations, and therefore, it is quite challenging to define health informatics career trends. Another recent study analyzed the content of US health care data scientist job postings to identify the required qualifications and skills for data scientist positions and emphasized the need for higher levels of education and skills training needed for health care data scientists [3].

In this study, we evaluated students’ perceived skills and self-confidence to develop health informatics apps. As professional and accrediting organizations have not determined distinct boundaries between competency and skill terms, we used these terms interchangeably. Currently, formal health informatics skills training with medical devices and apps are limited. Although students without any health informatics education might become an expert in programming, developing, and using an innovative health informatics app or system, others with formal education might not have any hands-on skills using the same apps. Hence, using competency and hands-on skills interchangeably in all cases is quite challenging. This research revealed that most students were employed (91/102, 89.2%), and presumably, they were knowledgeable about the required skills. Participants did not consider themselves experts in any skills, which indicates the need for the integration of skill-based training into the health informatics curriculum, except for skills in using Microsoft Word’s macro commands.

Furthermore, our research has several implications. First, this analysis identified a gap between existing competencies and in-demand skills. Developing innovative solutions to improve health care quality has become the major focus of leading health informatics companies, and recent publications emphasize that tomorrow’s workforce needs to design, develop, and implement innovative systems and work with new medical devices, patient monitoring apps, telemedicine, and smart home systems [5]. Owing to the lack of formal skills training, most health informaticians gain these practical experiences during their employment; thus, employers have been launching upskilling initiatives to keep their company competitive [44].

Second, this study revealed the need to determine new occupation-specific health informatics terms that will define different levels of practical know-how to generate disruptive ideas and design, develop, and implement sophisticated innovative technological solutions to health problems.

Third, we also identified the need to develop a specific competency assessment framework. Currently, AMIA uses Miller’s competency framework, which was initially developed to assess the clinical competence of medical school graduates. The Miller’s pyramid consists of four levels of clinical competence: knowing signs and symptoms (knows), knowing how to utilize exam and laboratory test data to diagnose a disease (knows how), demonstrate clinical performance (shows how), and being able to apply knowledge into practice (does) [45]. Although the Miller’s pyramid is widely accepted in medical practice to assess clinical competence, its application to health informatics has some limitations as this assessment model was not designed to assess any informatics competencies. As health informatics is an interdisciplinary field, the graduates might work in a wide range of settings and can follow different career paths, which makes the development of the competency framework extremely complex.

We developed a new framework that will include different tiers for evolving hands-on health informatics competencies (Figure 1). This competency framework divides the development of practical health informatics competencies into 6 hierarchical processes. The pyramid starts with knowledge acquisition at the bottom level. The next competence level is achieved when students acquire advanced hands-on health informatics skills to use specific computer software programs, sensor-based decision support systems, and other sophisticated patient monitoring apps. The third and fourth tiers represent applications of medical knowledge and technical knowledge. Although all health informaticians need to become familiar with these two competencies, teaching clinical health informatics tracks concentrating on the application of medical knowledge using health informatics systems and teaching nonclinical tracks concentrating on the application of technical knowledge such as programming, application of algorithmic principles, design, and development of mobile apps and other data science skills that we mentioned in our study might have more profound and meaningful outcomes. The fifth tier focuses on the application of problem-solving skills to manage and administer health informatics apps and programs. Finally, the sixth tier concentrates on innovative skills. Although most health informatics programs include capstone courses, these courses are usually designed to apply the knowledge gained through the master’s degree program rather than teaching new skill sets.

Health informatics specialists should know how to analyze and interpret health care data and identify potential areas of applications of AI. Defective AI algorithms can cause severe and unforeseen health consequences. However, integration of experiential AI training in health informatics curriculum and determination of necessary skill sets for different specializations is quite a challenging task as AI technology has many components such as machine learning, deep learning, pattern recognition, real-time data analytics, model building, data collection, and data visualization. This research demonstrated that participants defined themselves as an advanced beginner for skills in developing AI apps. This definition could be considered as being insufficient; however, a master’s degree program should consider students’ career perspectives and provide individualized tracks in addition to meeting mandatory accreditation standards. We propose that specific health informatics skills training should be identified using the enhanced health informatics curriculum components described in Figure 2 and be updated on a yearly basis.

Including R, Python, inferential and descriptive statistics, machine learning, database systems, and SQL, data presentation and visual encoding courses in the curriculum without real-life medical apps might not be enough to provide the required skills as students need to learn how to operate sophisticated medical equipment and remote monitoring devices and solve interoperability challenges. For example, with the hands-on laboratory exercises, students will be able to develop clinical decision support apps that can collect data from a wireless blood pressure monitor, wireless blood glucose meters, digital weight scales, and write the program code to integrate these apps with other databases. Consequently, depending on the students’ career plans, they might need further specialization such as integrating machine learning code with sophisticated medical software. Recent studies demonstrate the effectiveness of hands-on health informatics skills exercises [5,39].

Several limitations need to be acknowledged. This research was conducted in an academic institution, and the feedback was limited to the MS in health informatics students’ assessments. All students with different educational backgrounds were included in the study and analyzed together because skill-based training is not a part of the current curriculum. Thus, conducting regular national and international studies to analyze students’ confidence levels and course requests and comparing responses of students within the same educational backgrounds would be helpful.

The main objective of this study was to highlight evolving health informatics competencies rather than provide detailed information about country-level competencies. Owing to the universal nature of technology, health informaticians use the same data standards, methods, and algorithms to store, retrieve, and analyze the data around the world. Although national and international organizations have determined different foundational domains, professional and accrediting organizations have been updating their recommendations frequently and adopting similar measurable competencies (Figure 3) [13]. For instance, IMIA’s updated educational recommendations for nursing informatics and health informatics are the same [46]; conversely, the current studies emphasize the need for customization. We also observed that the existing literature and curriculum recommendations did not clearly delineate the difference between undergraduate and graduate health informatics competencies. As mentioned earlier, even though some recent publications assess health informatics training and identify universal competencies, there are still limited studies about skills training in the graduate health informatics curricula [47,48].

Health informatics graduates need hands-on experience with various health informatics tools and apps to develop skills and the ability to apply this practical expertise to unfamiliar situations, serve as subject-matter experts, and lead and manage innovative projects. Regional, national, and international accreditation standards, and in-demand technical skills to use and develop patient-centered health informatics systems could be taken into consideration when determining health informatics curriculum components. It is also essential to capture students’ perspectives before developing skills training components and to develop an up-to-date health informatics skills training framework depending on different medical specialties and health care needs for physicians, nurses, pharmacists, and medical and laboratory technologists. Developing new terminologies that will clearly specify the difference between competency-based and skill-based approaches for each health informatics discipline might be useful. Further studies that evaluate employers’ feedback and students’ perceptions after they get hired are suggested to determine the potential gaps and needs in health informatics skills training.