Published on in Vol 8, No 7 (2020): July

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/14795, first published .
Exploring the Determinants of Mobile Health Adoption by Hospitals in China: Empirical Study

Exploring the Determinants of Mobile Health Adoption by Hospitals in China: Empirical Study

Exploring the Determinants of Mobile Health Adoption by Hospitals in China: Empirical Study

Authors of this article:

Boumediene Ramdani1 Author Orcid Image ;   Binheng Duan2 Author Orcid Image ;   Ilhem Berrou3 Author Orcid Image

Original Paper

1Centre for Entrepreneurship, College of Business & Economics, Qatar University, Doha, Qatar

2Creative Assembly, Spire Court, Albion Way, Horsham, United Kingdom

3Faculty of Health and Applied Sciences, University of the West of England, Bristol, United Kingdom

Corresponding Author:

Boumediene Ramdani, PhD

Centre for Entrepreneurship

College of Business & Economics

Qatar University

PO Box 2713

Doha

Qatar

Phone: 974 44037762

Email: B.Ramdani@qu.edu.qa


Background: Although mobile health (mHealth) has the potential to transform health care by delivering better outcomes at a much lower cost than traditional health care services, little is known about mHealth adoption by hospitals.

Objective: This study aims to explore the determinants of mHealth adoption by hospitals using the technology-organization-environment (TOE) framework.

Methods: We conducted an interviewer-administered survey with 87 managers in Chinese public hospitals and analyzed the data using logistic regression.

Results: The results of our survey indicate that perceived ease of use (β=.692; P<.002), system security (β=.473; P<.05), top management support (β=1.466; P<.002), hospital size (β=1.069; P<.004), and external pressure (β=.703; P<.005) are significantly related to hospitals’ adoption of mHealth. However, information technology infrastructure (β=.574; P<.02), system reliability (β=−1.291; P<.01), and government policy (β=2.010; P<.04) are significant but negatively related to hospitals’ adoption of mHealth.

Conclusions: We found that TOE model works in the context of mHealth adoption by hospitals. In addition to technological predictors, organizational and environmental predictors are critical for explaining mHealth adoption by Chinese hospitals.

JMIR Med Inform 2020;8(7):e14795

doi:10.2196/14795

Keywords



Background

The aging population and the high prevalence of complex long-term conditions are placing unprecedented pressure on hospital services in China [1,2]. Mobile health (mHealth) not only has the potential to alleviate pressure on hospital services but can also increase accessibility and meet individual patient demands. It has been advocated as a complementary approach to traditional (ie, offline) health care services [3]. With over 1.3 billion mobile subscribers [4], mHealth services in China are considered the largest market in the world, accounting for 12.53 billion yuan or US $1.76 billion (1 yuan = US $0.14), in 2017 [5]. Defined as “the use of mobile devices—such as mobile phones, patient monitoring devices, personal digital assistants (PDAs) and wireless devices—for medical and public health practice” [6], mHealth has the potential to transform health care by delivering better outcomes at a lower cost [7]. For patients, mHealth has the potential to improve the health and well-being of individuals by recognizing behaviors, providing a rapid diagnosis of medical conditions, delivering just-in-time interventions, and continuous monitoring of their health status [8]. Recent evidence shows that mHealth could improve patient experience [9]. For health care providers, mHealth could reduce demands on clinicians’ time by minimizing office visits for the management of common conditions and enabling patient self-management [7].

China is a particularly interesting context for this study. In 2015, 1 in 4 persons aged ≥60 years lived in China, making it the largest population of older citizens in the world [10]. This trend is projected to grow by 71% between 2015 and 2030. Moreover, medical institutions in China are concentrated in cities, making it difficult to deliver health care services to the rural population [11]. Chinese policymakers need to address these challenges and find an effective solution that reaches both the elderly and rural populations. For that, mHealth is part of a national strategy to resolve the “difficulty and expense of seeking a doctor” [12].

Chinese hospitals have begun to take up mHealth to deliver health care services [3,9,12,13]. Most emerging literature focuses on patients’ adoption of mHealth services. However, little is known about the hospitals’ adoption of mHealth. Therefore, this study aims to address this gap by exploring the determinants of mHealth adoption by hospitals.

A large body of research has explored the determinants of adopting health care technologies. As highlighted in Table 1, previous research explored various health care technologies using several theoretical lenses in different settings. From reviewing the literature, it is still unclear what determines the adoption of mHealth by hospitals.

Table 1. Adoption of health care technologies.
Adoption theoryAdoption of technologyaConstructs/factorsbMethodDataLocation [reference]
TAMc with trust and perceived risksmHealthdPerceived usefulness, perceived ease of use, perceived risk, performance risk, legal concerns, and trustSEMe388 patients in large hospitalsChina [12]
TAM and UTAUTfMedical dashboard systemPerformance expectancy, effort expectancy, social influence, and facilitating conditionsSEM383 physicians and nurses in a tertiary teaching hospitalKorea [14]
TAM and UTAUTMobile electronic medical recordPerformance expectancy, effort expectancy, attitude, social influence, facilitating conditions, and behavior intention to useSEM449 subjects (65 physicians and 385 nurses) in a large tertiary hospitalKorea [15]
UTAUT 2mHealthPerformance expectancy, effort expectancy, social influence, facilitating conditions, hedonic motivation, price value, habit, waiting time, and self-conceptFactor analysis and path analysisA total of 3 surveys with 387, 359, and 375 patients who were offered mHealth as an alternative to traditional hospital servicesUnited States, Canada, and Bangladesh [16]
Task-technology fit and social contagion theoryEHRgAuthorization, compatibility, data quality, ease of use, information systems relationship, timeliness, locatability, system reliability, and social contagionSEMSurvey with 51 university students with working experience in the health care sector and used EHR systems in the pastUnited States [17]
UTAUTHome health care robotsPerformance expectancy, effort expectancy, social influence, facilitating conditions, trust, privacy concerns, ethical concerns, and legal concernsSEM and power analysis108 health care professionals and patients working for home health care agenciesUnited States [18]
Social capital theory, social cognitive theory and TAMTelehealthPerceived ease of use, perceived usefulness, system self-efficacy, social participation, institutional trust, and social trustSEM365 patients who used a telehealth system for at least one monthTaiwan [19]
TPBhmHealth servicePerceived value, attitude, perceived behavior control, subjective norm, perceived physical condition, resistance to change, technology anxiety, and self-actualization needSEM424 middle-aged and older people accessing community service centersChina [20]
UTAUTHITiPerformance expectancy, effort expectancy, social influence, facilitating conditions, and provincial areasSEM400 health care professionals working in hospitalThailand [21]
No specific theorymHealth usage intention, assimilation, and channel preferencesIndividual difference, health care availability and health care utilization, and socioeconomic status and demographicsHierarchical ordinary least squares1132 consumersUnited States [22]
Decomposed TPB and value-attitude-behaviorMEDLINE systemPerceived usefulness, perceived ease of use, attitude, interpersonal influence, subjective norm, personal innovativeness in ITj, self-efficacy, facilitating conditions, perceived behavioral control, and usage intentionSEM224 physicians in primary care centers and hospitalsTaiwan [23]
TAM and TPBMobile health careAttitude, perceived behavioral control, subjective norm, perceived usefulness, perceived ease of use, personal innovativeness, and perceived service availabilitySEM140 health care professionals working in hospitalsTaiwan [24]
UTAUTHITPerformance expectancy, effort expectancy, social influence, voluntariness, facilitating conditions, experience, and IT knowledgeSEMInformation management officers or head officers from 1323 community health centersThailand [25]
TAM and innovation diffusion theoryElectronic logistics information systemCompatibility, perceived usefulness, perceived ease of use, trust, perceived financial cost, and behavioral intentionSEMNurses working in 10 hospitals who used electronic logistics information systemTaiwan [26]
TAMTelemedicinePerceived usefulness and perceived ease of useSEM408 physicians working in tertiary hospitalsHong Kong [27]

aDependent variable.

bIndependent variables.

cTAM: technology acceptance model.

dmHealth: mobile health.

eSEM: structural equation modelling.

fUTAUT: unified theory of acceptance and use of technology.

gEHR: electronic health record.

hTPB: theory of planned behavior.

iHIT: health information technology.

jIT: information technology.

Research Model and Hypotheses

The technology-organization-environment (TOE) framework, first introduced by Tornatzky and Fleischer [28], has been used as a guiding theoretical basis for the adoption of many technologies. This framework integrates the 3 vital contexts—technology, organization, and environment—to provide a holistic understanding of adoption of technology from an organizational perspective. The generic nature of the framework allows researchers to explore the determinants that are relevant to their specific context. Moreover, it has the empirical support of several studies exploring the adoption of technology in different types of organizations and different sectors of the economy (Table 2). A number of technologies have been examined from open systems and electronic business (e-business) to enterprise systems, radio-frequency identification (RFID), and cloud computing. Most of these studies focused on the adoption of technology from an organizational perspective, including government agencies [29], large firms [30-32], and small-to-medium–sized enterprises (SMEs) [33-37]. Other studies focused on particular sectors, including hotels [38], retailing [39], manufacturing [38,40], and the services sector [40]. Surprisingly, only two studies addressed the adoption of technology from a hospital perspective [41,42]. This study aims to extend this body of research by using the TOE model to explore the determinants of mHealth adoption by hospitals.

From reviewing the literature, we developed a TOE framework to explore mHealth adoption by hospitals in China. Our framework (Figure 1) suggests that hospital adoption of mHealth is influenced by TOE determinants: technology—perceived usefulness, perceived ease of use, system compatibility, system security, and information technology (IT) infrastructure; organization—top management support (TMS), organizational readiness, and size; and environment—government policy and regulations, and external pressure. Each of the 3 contexts is discussed below to develop our hypotheses.

Table 2. Technology-organization-environment determinants of adoption of technology.
Adoption of technologya [reference]Determinantsb

TechnologyOrganizationEnvironment

Perceived usefulnessPerceived ease of useCompatibilitySystem securityITc infrastructureTop management supportOrganizational readinessSizeGovernment policyExternal pressure
Cloud computing [29]N/AdN/AN/AN/AXeXN/AN/AXX
e-SCMf [30]XN/AN/AN/AN/AXXN/AN/AX
Open systems [32]N/AN/AN/AN/AXN/AN/AN/AN/AX
Electronic data interchange [33]XN/AN/AN/AXN/AN/AN/AXX
Internet [34]XN/AN/AN/AXN/AN/AN/AN/AN/A
Enterprise systems [35]XN/AN/AN/AN/AXXXN/AN/A
Enterprise applications [36]XXXN/AN/AXXXN/AX
Enterprise resource planning [37]XN/AXXXN/AN/AXXX
Mobile reservation systems [38]XXXN/AXXN/AXN/AX
e-Businessg [39]N/AN/AN/AN/AXN/AN/AXXX
Cloud computing [40]XXN/AN/AN/AXN/AXN/AN/A
Health information exchange [41]N/AN/AN/AN/AXN/AN/AN/AN/AX
RFIDh [42]XN/AXXN/AXXN/AXX
Cloud computing [43]N/AXN/AN/AXN/AN/AN/AN/AX
RFID [44]XN/AN/AN/AN/AXN/AXN/AN/A
RFID [45]XXXN/AXXN/AXN/AX
e-Business [46]XN/AXXXN/AN/AXN/AX
e-Business [47]N/AN/AN/AN/AXXN/AXXX
e-Commercei [48]XN/AN/AN/AN/AN/AN/AN/AXX
e-Commerce [49]N/AN/AN/AN/AXN/AN/AN/AXX

aDependent variable.

bIndependent variables.

cIT: information technology.

dN/A: not applicable.

eX: significant determinant.

fe-SCM: electronic supply chain management.

ge-Business: electronic business.

hRFID: radio-frequency identification.

ie-Commerce: electronic commerce.

Figure 1. Technological organizational and environmental determinants of mHealth adoption by hospitals.
View this figure
Technology

A number of technological determinants have been shown to affect organizational adoption of new technologies, including perceived usefulness, perceived ease of use, system compatibility, system security, and IT infrastructure. New technology is more likely to be adopted when an organization perceives it to offer more benefits than existing systems. Several new technologies have proven their advantage over existing practices, including e-business over physical stores [31,46], RFID over manual data entry [44,45], cloud computing over client-server computing [40], and mobile reservation over the telephone or web-based reservations [38]. In the health care context, the perceived benefits of RFID in keeping track of hospital patients were found to be a significant determinant of adoption [42]. In their study of customer relationship management (CRM), Hung et al [23] also found a relative advantage to be positively associated with CRM adoption by hospitals. On this basis, we suggested the following:

Hypothesis 1: Perceived usefulness will positively influence hospitals’ adoption of mHealth.

Perceived ease of use has been found to be a significant factor in the adoption of technology [36,38,40,43,45]. The widespread use of smartphones and health monitoring devices has made it easier for consumers to handle such devices remotely [16]. For hospitals to have a more active engagement in health care delivery, we expect the perceived ease of use of mobile technologies to be positively associated with mHealth adoption. Thus, we proposed the following:

Hypothesis 2: Perceived ease of use will positively influence hospitals’ adoption of mHealth.

System compatibility has been found as a significant determinant for the adoption of technology [36-38,45,46]. In the health care context, compatibility with a non-RFID–based patient tracking system was found to be a crucial determinant for RFID application integration [42]. Therefore, we suggested the following hypothesis:

Hypothesis 3: System compatibility with existing systems will positively influence hospitals’ adoption of mHealth.

Unauthorized data access is a concern for organizations and their clients and has the potential to jeopardize information security and privacy [50]. Similar to e-business, mobile technologies are integrated into transactions that involve fund transfer and the exchange of organizational data [46]. Although a few studies [37,46] found system security to be positively associated with the adoption of technology, it is of particular significance for health care providers. Security and privacy protection were found to be major determinants of RFID adoption by hospitals [42].A security breach of patient information not only puts patients at risk but can also cause a lasting damage to a hospital’s reputation. Thus, we proposed the following:

Hypothesis 4: System security will positively influence hospitals’ adoption of mHealth.

Finally, IT infrastructure has been found to be one of the most significant determinants of the adoption of technology. This factor was found to be significant for most types of technologies, including open systems [32], e-business [31,46], RFID [45], enterprise resource planning [37], mobile reservation systems [38], and cloud computing [29]. In a study of health information exchange (HIE), hospitals that do not have the necessary technological infrastructure were found to be less likely to adopt new systems [41]. Therefore, we expect IT infrastructure to be a significant determinant of mHealth adoption by hospitals, and we proposed the following hypothesis:

Hypothesis 5: IT infrastructure will positively influence hospitals’ adoption of mHealth.
Organization

TMS, organizational readiness, and organizational size have been found to influence the organizational adoption of new technologies. TMS has been found to be one of the most significant determinants of the adoption of technology [51] as managers can overcome barriers to adoption and resistance to change [30]. It has been found to be an influential factor in the adoption of e-business [46], enterprise systems [35], RFID [44,45], and mobile reservation systems [38]. In hospitals, TMS has been shown to be a significant determinant of RFID adoption for patient tracking [42]. Thus, we suggested the following:

Hypothesis 6: TMS will positively influence hospitals’ adoption of mHealth.

Organizational readiness, the degree to which an organization has the knowledge and resources that can remove barriers to system adoption [52], has been shown to be positively related to the adoption of new technology [30]. It has been found to influence the adoption of enterprise systems significantly [35,36]. In the health care context, organizational readiness has been shown to be of critical importance in RFID adoption [42]. Therefore, we proposed the following:

Hypothesis 7: Organizational readiness positively influences hospitals’ adoption of mHealth.

Organizational size is another important determinant of adoption of technology [51]. It was established that larger firms are more likely to adopt enterprise systems [35-37]. Other studies also confirm the significance of organizational size [38-40,45-47]. Thus, we suggested the following:

Hypothesis 8: Hospital size will positively influence hospitals’ adoption of mHealth.
Environment

Government policy and external pressure have been shown to influence the organizational adoption of new technologies. Existing laws and regulations can critically impact the adoption of new technologies [29,37]. In the health care context, compliance with legislation and standards has been found to be critical in the adoption of RFID patient tracking [42]. Government regulation has been advocated to play a more important role in the Chinese context compared with other developed economies [49]. As hospitals in China are state-owned, government policy can encourage or discourage hospitals from adopting mHealth. On this basis, we proposed the following:

Hypothesis 9: Government policy will positively influence hospitals’ adoption of mHealth.

External pressure has been found to be one of the most significant determinants of organizational adoption of new technologies [51]. Health care organizations are under constant pressure to adopt and implement new technologies to become more efficient [53]. To coordinate care and steer patients away from emergency departments, hospitals are under pressure to adopt HIE [41]. Thus, we suggested the following:

Hypothesis 10: External pressure will positively influence hospitals’ adoption of mHealth.

Data Collection

An interviewer-administered survey was conducted to test the research model empirically. The questionnaire was piloted and refined through rigorous pretesting. In this phase, 8 public hospital managers were invited to participate in this study and comment on instrument clarity and question wording. As a result, construct measures were revised. In addition, system reliability [54] was emphasized by hospital directors as a missing variable that should be included in the research instrument. Here, we revised the research instrument to include system reliability as a further measure of the technology context. A company would choose not to adopt cloud computing because of the increased business risk associated with the uncertainty of service availability and reliability, especially if there are unexpected downtimes and disruptions [55]. People often would not prefer to use new technology because of concerns about the reliability and stability of the system [55]. System reliability is key when providing uninterrupted services as shown in a study by Pagani [56]. Thus, we expect system reliability to influence hospitals’ adoption of mHealth significantly.

A convenient snowballing approach was used in this study. Due to the difficulty in obtaining data from public hospitals in China, the authors focused on collecting data from 2 regions, namely Shanghai and Gansu. A total of 91 questionnaires were obtained, but 4 questionnaires were discarded because of missing data. As a result, 87 responses were included in the analysis. Respondents, from hospitals that are not willing to adopt mHealth, were classified as nonadopters, whereas respondents from hospitals that are willing to adopt mHealth in the next 3 years were classified as adopters. Table 3 shows the characteristics of the responding hospitals in terms of location, hospital level, hospital beds, and respondents’ positions. Hospitals in China are classified into 3 major tiers, with 3 subtiers within each major tier [29,57]. Level 3 hospitals provide specialized medical services in several departments, with level 3A hospitals being the most advanced. These hospitals have a minimum capacity of 500 beds. Level 2 hospitals are regional hospitals with 100 to 499 beds providing cross-community medical services. These are smaller in size and less advanced than those in level 3 hospitals. Level 1 hospitals provide basic health care facilities with a capacity of 20 to 99 beds.

The questionnaire was translated into Chinese official language (standard Mandarin) following the conventional forward-then-back-translation approach. This has taken into account local culture and dialect considerations when establishing conceptual equivalence between English and Chinese versions of the instrument [58].

Table 3. Hospitals’ and respondents’ characteristics.
DemographicsAdopters (n=50), n (%)Nonadopters (n=37), n (%)
Location

Shanghai37 (74)9 (24)

Gansu13 (26)28 (76)
Public hospital level

<Level 335 (70)7 (19)


Level 3A29 (58)5 (14)


Level 3B6 (12)2 (5)

>Level 315 (30)30 (81)


Level 212 (24)24 (65)


Level 13 (6)6 (16)
Hospitals beds

500+35 (70)7 (19)

100-49912 (24)24 (65)

20-993 (6)6 (16)
Respondents’ job titles

Directors of laboratory services37 (74)10 (27)

Directors of ITa department6 (12)9 (24)

Directors of other departments7 (14)18 (49)

aIT: information technology.

Measures

Measurement items were developed based on a comprehensive review of the literature and modified to suit the mHealth context in China. All measurement items are listed in Multimedia Appendix 1. The items for perceived usefulness, perceived ease of use, and system compatibility were adopted from Moore and Benbasat [59]. The measure for system security was developed from Kim et al [60], the measure for IT infrastructure was developed from Bhattacherjee and Hikmet [61], and the measure for system reliability was adapted from Goodhue and Thompson [54]. The items for TMS were adapted from Yap et al [62], and the items for organizational readiness were adapted from Grandon and Pearson [63]. The items for government policy were adapted from Chau and Tam [32], and the items for external pressure were adopted from Premkumar and Roberts [64]. A 5-point Likert scale ranging from strongly disagree to strongly agree was used for all measurement items with the exception of hospital size, which was classified into 3 major tiers [29,57].


Validity and Reliability

The validity of construct measures was assessed using principal component analysis with orthogonal factor rotation. All factor loadings were above 0.5 [65]. Reliability was assessed using Cronbach α. All α coefficients exceeded .7 [65]. As shown in Table 4, factor analysis and α coefficient results indicate adequate validity and reliability of the measures.

The correlation matrix was examined for multicollinearity problems. Table 5 shows that none of the squared correlation coefficients are above the 0.9 level [65]. Table 6 shows that the variance inflation factor values are not greater than the cutoff value of 10 [65], indicating that multicollinearity is not a problem for this study.

Table 4. Factor analysis and reliability assessment.
Constructs and itemsPUaPEbSCcSSdITIeSRfTMSgORhGPiEPj
PU1.603k
PU2.825
PU3.727
PU4.878
PU5.584
PU6.821
PE1.621
PE2.820
PE3.773
PE4.631
PE5.739
PE6.707
SC1.694
SC2.836
SC3.663
SS1.930
SS2.896
SS3.832
ITI1.688
ITI2.859
ITI3.721
SR1.821
SR2.956
SR3.772
TMS1.719
TMS2.819
TMS3.897
OR1.877
OR2.859
OR3.628
GP1.873
GP2.880
EP1.702
EP2.823
EP3.895
Eigenvalue2.7484.3304.0392.6532.3672.6862.2502.1521.7111.355
Variance7.63412.02811.2217.3696.5767.4606.2515.9784.7523.764
Cronbach α.778.778.786.797.794.778.764.778.790.777

aPU: perceived usefulness.

bPE: perceived ease of use.

cSC: system compatibility.

dSS: system security.

eITI: information technology infrastructure.

fSR: system reliability.

gTMS: top management support.

hOR: organizational readiness.

iGP: government policy.

jEP: external pressure.

kConstructs and items.

Table 5. Correlations between independent variables.
ConstructsPUaPEbSCcSSdITIeSRfTMSgORhHSiGPjEPk
PU1.000N/AlN/AN/AN/AN/AN/AN/AN/AN/AN/A
PE−0.2771.000N/AN/AN/AN/AN/AN/AN/AN/AN/A
SC−0.2470.3941.000N/AN/AN/AN/AN/AN/AN/AN/A
SS0.0050.5120.2781.000N/AN/AN/AN/AN/AN/AN/A
ITI0.054−0.440−0.366−0.3681.000N/AN/AN/AN/AN/AN/A
SR−0.054−0.734−0.481−0.4970.5071.000N/AN/AN/AN/AN/A
TMS−0.2350.6590.4890.454−0.594−0.6941.000N/AN/AN/AN/A
OR.0480.4560.4400.476−0.496−0.6120.5161.000N/AN/AN/A
HS−0.1220.5310.3840.471−0.202−0.5540.5670.2691.000N/AN/A
GP0.250−0.786−0.579−0.5750.4700.708−0.824−0.634−0.5621.000N/A
EP−0.0980.4690.3320.399−0.207−0.4960.5120.1870.575−0.6431.000

aPU: perceived usefulness.

bPE: perceived ease of use.

cSC: system compatibility.

dSS: system security.

eITI: information technology infrastructure.

fSR: system reliability.

gTMS: top management support.

hOR: organizational readiness.

iHS: hospital size.

jGP: government policy.

kEP: external pressure.

lN/A: not applicable.

Table 6. Collinearity statistics.
ConstructsToleranceVariance inflation factor
Perceived usefulness0.6381.567
Perceived ease of use0.5281.896
System compatibility0.7351.360
System security0.8411.188
Information technology infrastructure0.7421.347
System reliability0.5671.764
Top management support0.4752.106
Organizational readiness0.5641.773
Hospital size0.7421.347
Government policy0.5641.773
External pressure0.5431.840

Model Testing

Logistic regression was used as the dependent variable was dichotomous (nonadopters vs adopters). This technique has been utilized in previous studies on the organizational adoption of technologies such as mobile reservation systems [38], electronic supply chain management [30], and enterprise systems [35].

Table 7 shows the results of the logistic regression analysis. The chi-square test was significant (omnibus X211=70.4; P<.001), and 2 pseudo R2 values (Cox and Snell R2=0.55; Nagelkerke R2=0.74) were satisfactory. Moreover, the research model correctly predicted 81% (30/37) of the nonadopters and 88% (44/50) of the adopters with an overall predictive accuracy of 85% (Table 8). Overall, the research model exhibits an acceptable fit with the data.

Table 7. Results of the logistic regression.
Constructsaβ coefficientWald statisticP value
Perceived usefulness−.0960.424.51
Perceived ease of use.692b9.406.002
System compatibility.5613.083.07
System security.473c3.828.05
Information technology infrastructure−.574c4.784.02
System reliability−1.291c6.123.01
Top management support1.466b9.614.002
Organizational readiness.6052.170.14
Hospital size1.069b8.345.004
Government policy−2.010c6.516.04
External pressure.703b3.972.005

aGoodness-of-fit: omnibus X211=70.4; P<.001; −2 log likelihood value=118.658; Cox and Snell R2=0.55; Nagelkerke R2=0.74.

bP<.01.

cP<.05.

Table 8. Discriminating power.
ObservedPredictedPercentage correct

Nonadopters, n (%)Adopters, n (%)
Nonadopters30 (81)7 (19)81
Adopters6 (12)44 (88)88
OverallN/AaN/A85

aN/A: not applicable.

As shown in Table 7, perceived ease of use (β=.692; P<.002), system security (β=.473; P<.05), TMS (β=1.466; P<.002), hospital size (β=1.069; P<.004), and external pressure (β=.703; P<.005) were significantly related to hospitals’ adoption of mHealth. IT infrastructure (β=−.574; P<.02), system reliability (β=−1.291; P<.01), and government policy (β=−2.010; P<.04) were significant but negatively related to hospitals’ adoption of mHealth. Thus, hypotheses 2, 4, 6, 8, and 10 are supported. However, perceived usefulness, system compatibility, and organizational readiness did not exhibit a significant relationship with hospitals’ adoption of mHealth. Thus, hypotheses 1, 3, and 7 are not supported. These findings are summarized in Table 9.

Table 9. Summary of hypotheses support.
PredictorsHospital mobile health adoption
Perceived usefulnessReject
Perceived ease of useAccept
System compatibilityReject
System securityAccept
Information technology infrastructureReject (significant but negative)
System reliabilityReject (significant but negative)
Top management supportAccept
Organizational readinessReject
Hospital sizeAccept
Government policyReject (significant but negative)
External pressureAccept

Principal Findings

This study explores the determinants of hospitals’ adoption of mHealth using the TOE framework. The technological determinants of mHealth adoption by hospitals were examined. Although perceived ease of use and system security are facilitators, IT infrastructure and system reliability are inhibitors of mHealth adoption by hospitals. Perceived ease of use has been found to be a significant determinant of mHealth adoption not only among diabetic patients [66] but also among health care professionals [67]. In addition, security and privacy protection has been found to influence hospital adoption of mHealth [66] and RFID patient tracking [42]. Lack of security was found to be a barrier to telemedicine adoption by physicians [68]. Unexpectedly, IT infrastructure and system reliability were significant but negatively related to hospitals’ adoption of mHealth. These results differ from those obtained by Vest [41], who noted that hospitals with low levels of IT infrastructure readiness have lower odds of HIE adoption. They also differ from the results obtained by Shareef et al [66], who found that perceived reliability is positively associated with mHealth adoption. A possible explanation for the negative relationships is the lack of a comprehensive strategy to invest, implement, and use mHealth. Evidence indicates that even among hospitals with established strategies to adopt mHealth solutions, only a few attempt to integrate and align these mHealth solutions with their existing IT systems [69].

Surprisingly, perceived usefulness does not exhibit a positive effect on hospitals’ adoption of mHealth. The reason for this insignificant result could be because of the lack of awareness of the benefits of adopting mHealth solutions. This finding is similar to that of Wang et al [45] study of RFID adoption by manufacturing firms and the study [38] of adoption of mobile reservation systems by hotels. In addition, system compatibility does not exhibit a positive effect on hospitals’ adoption of mHealth. Here, hospital managers seem to underestimate the significance of system compatibility and the extent to which mHealth is perceived to be consistent with their needs, values, and experiences. A possible explanation for this insignificant result is that new mHealth technologies can be easily integrated with existing systems. This finding is similar to that of both study of enterprise systems adoption by SMEs by Ramdani et al [35] and the study of cloud computing adoption in manufacturing and services firms by Oliveira et al [40].

Organizational determinants of mHealth adoption by hospitals were investigated. As expected, TMS and hospital size are facilitators of mHealth adoption by hospitals. These findings are consistent with Cao et al [42], who found management support to be key to the success of RFID application in hospitals, and Hung et al [23], who found hospital size to be a critical factor in CRM adoption by hospitals. Surprisingly, organizational readiness does not exhibit a significant effect on hospitals’ adoption of mHealth. Although lack of financing has been found to be a barrier to the adoption of 3 health information technologies, including electronic health record functionalities, electronic-prescriptions, and telemedicine [68], the reason behind the insignificance of organizational readiness is that mHealth technologies do not require a substantial upfront investment. The cost associated with mHealth tends to be much lower than that of traditional medical services [12].

The environmental determinants of mHealth adoption by hospitals were investigated. Although government policy is an inhibitor, external pressure is a facilitator of mHealth adoption by hospitals. Government policy is significant but negatively associated with hospitals’ adoption of mHealth. The lack of an enabling health care policy has been suggested as a barrier to mHealth adoption [70]. Furthermore, the current highly regulated environment in hospitals in China could hinder the adoption of new technologies. As expected, external pressure is positively associated with hospital adoption of mHealth. This result is supported by Cao et al [42] study of RFID adoption in the health care sector, where external pressure was found to be one of the key dimensions of the environmental context. Moreover, competition between health care organizations has been found to be a key determinant of HIE adoption [41].

Study Implications and Limitations

The results of this study provide practical implications for mHealth suppliers and policymakers. First, both perceived ease of use and system security in the technological context have a significant effect on hospitals’ adoption of mHealth. To facilitate hospital adoption, mHealth developers and suppliers need to ensure that the adoption process is relatively simple, and the system is highly secure. Second, both TMS and hospital size in the organizational context have a significant positive effect on hospitals’ adoption of mHealth. To get hospitals to adopt mHealth, suppliers need to direct their advertising and promotions toward senior executives who make the final decision to adopt. In addition, mHealth suppliers may need to target larger hospitals because they are likely to invest in such systems. Finally, government policy and external pressure in the environmental context have a significant effect on hospitals’ adoption of mHealth. Although external pressure facilitates the adoption of mHealth, existing government policies must be revised to encourage the adoption of new technologies in the health care sector.

Several potential limitations must be considered when interpreting the results of this study. First, this study focuses only on hospitals’ adoption of mHealth. The impact of mHealth on hospital performance should be examined to gain a holistic understanding. Second, a set of technological, organizational, and environmental predictors were examined. Future studies may examine whether other predictors may influence hospitals’ adoption of mHealth. Third, the collected data are cross-sectional and posited causal relationships could only be inferred. Future studies could collect longitudinal data to determine causal links more explicitly. Fourth, the statistical technique employed (ie, logistic regression) only analyses the relationships between hospitals' adoption of mHealth and their predictors and does not analyze the relationships between the predictors. Future studies could use other statistical techniques to examine the relationships between the predictors and elaborate on the findings of this study. Another important limitation lies in the sample size and type of hospitals studied. Although our sample size was adequate for this study, the findings might vary with larger samples. In addition, because private hospitals are developing rapidly in China, it will be worth examining how our findings compare with private hospitals’ adoption of mHealth. Finally, our data are largely dominated by hospitals from 2 regions in China: Shanghai and Gansu. We have overlooked the potential regional differences in our study. Thus, the research model should be tested further using larger samples from other regions or even from other counties to make cross-region or cross-country comparisons.

Conclusions

This study contributes to the literature on organizational mHealth adoption by examining the determinants of mHealth adoption by hospitals. The results indicate that significant predictors of hospitals' adoption of mHealth include perceived ease of use, system security, IT infrastructure, system reliability, TMS, hospital size, external pressure, and government policy. However, perceived usefulness and system compatibility in the technological context and organizational readiness in the organizational context are not significant predictors.

The contributions of this study to research on organizational mHealth adoption are 3-fold. First, previous studies focused on the adoption of mHealth at an individual level, including health care professionals and patients. This study adds important insights to the literature by focusing on the organizational (ie, hospital) adoption of mHealth. Second, limited knowledge exists on the adoption of technology in Chinese health care organizations. This study contributes to the literature by highlighting the context-specific determinants of mHealth adoption. Third, studies of adoption of technology in health care organizations mainly use versions of the unified theory of acceptance and use of technology framework. This study uses the TOE framework to contribute to the adoption of technology in the health care literature by identifying the predictors that influence hospitals to adopt mHealth.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Questionnaire: Measurement of constructs.

DOCX File , 14 KB

  1. Zhang X, Lai K, Guo X. Promoting China's mhealth market: a policy perspective. Health Policy Technol 2017 Dec;6(4):383-388. [CrossRef]
  2. Sun J, Guo Y, Wang X, Zeng Q. mHealth for aging China: opportunities and challenges. Aging Dis 2016 Jan;7(1):53-67 [FREE Full text] [CrossRef] [Medline]
  3. Meng F, Guo X, Peng Z, Lai K, Zhao X. Investigating the adoption of mobile health services by elderly users: trust transfer model and survey study. JMIR Mhealth Uhealth 2019 Jan 8;7(1):e12269 [FREE Full text] [CrossRef] [Medline]
  4. International Telecommunications Union. 2016. ​​​​​​ITU Estimates That at the End of 2019, 53.6 Per Cent of the Global Population, or 4.1 Billion People, Are Using the Internet   URL: http://www.itu.int/en/ITU-D/Statistics/Pages/stat/default.aspx [accessed 2020-05-11]
  5. GSMA. 2018. Review of China Mobile Health Market and Outlook for Future   URL: https:/​/www.​gsma.com/​iot/​/wp-content/uploads/2013/03/1.​-Mr-%E5%BC%A0%E6%AF%85-ZhangYi-ii-Research-Review-of-China-Mobile-Health-Market-and-Outlook-for-Future.​pdf [accessed 2019-05-23]
  6. World Health Organization. Global Diffusion of eHealth: Making Universal Health Coverage Achievable. Report of the Third Global Survey on eHealth. Geneva, Switzerland: World Health Organization; 2016.
  7. Steinhubl SR, Muse ED, Topol EJ. Can mobile health technologies transform health care? J Am Med Assoc 2013 Dec 11;310(22):2395-2396. [CrossRef] [Medline]
  8. Kumar S, Nilsen W, Pavel M, Srivastava M. Mobile health: revolutionizing healthcare through transdisciplinary research. Comput 2013 Jan;46(1):28-35. [CrossRef]
  9. Lu C, Hu Y, Xie J, Fu Q, Leigh I, Governor S, et al. The use of mobile health applications to improve patient experience: cross-sectional study in Chinese public hospitals. JMIR Mhealth Uhealth 2018 May 23;6(5):e126 [FREE Full text] [CrossRef] [Medline]
  10. United Nations. 2015. World Population Ageing   URL: http://www.un.org/en/development/desa/population/publications/pdf/ageing/WPA2015_Report.pdf [accessed 2020-05-11]
  11. Ni Z, Wu B, Samples C, Shaw R. Mobile technology for health care in rural China. Int J Nurs Stud 2014 Sep;1(3):323-324 [FREE Full text] [CrossRef]
  12. Deng Z, Hong Z, Ren C, Zhang W, Xiang F. What predicts patients' adoption intention toward mhealth services in China: empirical study. JMIR Mhealth Uhealth 2018 Aug 29;6(8):e172 [FREE Full text] [CrossRef] [Medline]
  13. Ye Q, Deng Z, Chen Y, Liao J, Li G, Lu Y. How resource scarcity and accessibility affect patients' usage of mobile health in China: resource competition perspective. JMIR Mhealth Uhealth 2019 Aug 9;7(8):e13491 [FREE Full text] [CrossRef] [Medline]
  14. Lee K, Jung SY, Hwang H, Yoo S, Baek HY, Baek R, et al. A novel concept for integrating and delivering health information using a comprehensive digital dashboard: An analysis of healthcare professionals' intention to adopt a new system and the trend of its real usage. Int J Med Inform 2017 Jan;97:98-108. [CrossRef] [Medline]
  15. Kim S, Lee K, Hwang H, Yoo S. Analysis of the factors influencing healthcare professionals' adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital. BMC Med Inform Decis Mak 2016 Jan 30;16:12 [FREE Full text] [CrossRef] [Medline]
  16. Dwivedi Y, Shareef M, Simintiras A, Lal B, Weerakkody V. A generalised adoption model for services: a cross-country comparison of mobile health (m-health). Gov Inf Q 2016 Jan;33(1):174-187 [FREE Full text] [CrossRef]
  17. Gan Q. Is the adoption of electronic health record system 'contagious'? Health Policy and Technol 2015 Jun;4(2):107-112 [FREE Full text] [CrossRef]
  18. Alaiad A, Zhou L. The determinants of home healthcare robots adoption: an empirical investigation. Int J Med Inform 2014 Nov;83(11):825-840. [CrossRef] [Medline]
  19. Tsai C. Integrating social capital theory, social cognitive theory, and the technology acceptance model to explore a behavioral model of telehealth systems. Int J Environ Res Public Health 2014 May 7;11(5):4905-4925 [FREE Full text] [CrossRef] [Medline]
  20. Deng Z, Mo X, Liu S. Comparison of the middle-aged and older users' adoption of mobile health services in China. Int J Med Inform 2014 Mar;83(3):210-224. [CrossRef] [Medline]
  21. Phichitchaisopa N, Naenna T. Factors affecting the adoption of healthcare information technology. Excli J 2013;12:413-436 [FREE Full text] [Medline]
  22. Rai A, Chen L, Pye J, Baird A. Understanding determinants of consumer mobile health usage intentions, assimilation, and channel preferences. J Med Internet Res 2013 Aug 2;15(8):e149 [FREE Full text] [CrossRef] [Medline]
  23. Hung S, Ku Y, Chien J. Understanding physicians' acceptance of the medline system for practicing evidence-based medicine: a decomposed TPB model. Int J Med Inform 2012 Feb;81(2):130-142. [CrossRef] [Medline]
  24. Wu L, Li J, Fu C. The adoption of mobile healthcare by hospital's professionals: an integrative perspective. Decis Support Syst 2011 Jun;51(3):587-596 [FREE Full text] [CrossRef]
  25. Kijsanayotin B, Pannarunothai S, Speedie SM. Factors influencing health information technology adoption in Thailand's community health centers: applying the UTAUT model. Int J Med Inform 2009 Jun;78(6):404-416. [CrossRef] [Medline]
  26. Tung F, Chang S, Chou C. An extension of trust and TAM model with IDT in the adoption of the electronic logistics information system in HIS in the medical industry. Int J Med Inform 2008 May;77(5):324-335. [CrossRef] [Medline]
  27. Hu P, Chau P, Sheng O, Tam K. Examining the technology acceptance model using physician acceptance of telemedicine technology. J Manage Inform Syst 2015 Dec 2;16(2):91-112 [FREE Full text] [CrossRef]
  28. Tornatzky LG, Fleischer M. The Processes of Technological Innovation. Maryland, United States: Lexington Books; 1990.
  29. Liang Y, Qi G, Wei K, Chen J. Exploring the determinant and influence mechanism of e-government cloud adoption in government agencies in China. Gov Inf Q 2017 Sep;34(3):481-495 [FREE Full text] [CrossRef]
  30. Lin H. Understanding the determinants of electronic supply chain management system adoption: using the technology–organization–environment framework. Technol Forecast Soc 2014 Jul;86:80-92 [FREE Full text] [CrossRef]
  31. Lin H, Lin S. Determinants of e-business diffusion: a test of the technology diffusion perspective. Technovation 2008 Mar;28(3):135-145 [FREE Full text] [CrossRef]
  32. Chau P, Tam K. Factors affecting the adoption of open systems: an exploratory study. MIS Q 1997 Mar;21(1):1 [FREE Full text] [CrossRef]
  33. Kuan K, Chau P. A perception-based model for EDI adoption in small businesses using a technology–organization–environment framework. Inf Manag 2001 Oct;38(8):507-521 [FREE Full text] [CrossRef]
  34. Alam SS. Adoption of internet in Malaysian SMEs. J Small Bus Enterprise Dev 2009 May 15;16(2):240-255 [FREE Full text] [CrossRef]
  35. Ramdani B, Chevers D, Williams D. SMEs' adoption of enterprise applications: a technology-organisation-environment model. J Small Bus Enterprise Dev 2013;20(4):735-753 [FREE Full text] [CrossRef]
  36. Ramdani B, Kawalek P, Lorenzo O. Predicting SMEs' adoption of enterprise systems. J Enterprise Inf Manag 2009 Feb 13;22(1/2):10-24 [FREE Full text] [CrossRef]
  37. Awa H, Ojiabo O. A model of adoption determinants of ERP within T-O-E framework. Inf Technol People 2016 Nov 7;29(4):901-930 [FREE Full text] [CrossRef]
  38. Wang Y, Li H, Li C, Zhang D. Factors affecting hotels' adoption of mobile reservation systems: a technology-organization-environment framework. Tour Manag 2016 Apr;53:163-172 [FREE Full text] [CrossRef]
  39. Zhu K, Kraemer K. Post-adoption variations in usage and value of e-business by organizations: cross-country evidence from the retail industry. Inf Syst Res 2005 Mar;16(1):61-84 [FREE Full text] [CrossRef]
  40. Oliveira T, Thomas M, Espadanal M. Assessing the determinants of cloud computing adoption: an analysis of the manufacturing and services sectors. Inf Manag 2014 Jul;51(5):497-510 [FREE Full text] [CrossRef]
  41. Vest JR. More than just a question of technology: factors related to hospitals' adoption and implementation of health information exchange. Int J Med Inform 2010 Dec;79(12):797-806. [CrossRef] [Medline]
  42. Cao Q, Jones D, Sheng H. Contained nomadic information environments: technology, organization, and environment influences on adoption of hospital RFID patient tracking. Inf Manag 2014 Mar;51(2):225-239 [FREE Full text] [CrossRef]
  43. Gutierrez A, Boukrami E, Lumsden R. Technological, organisational and environmental factors influencing managers’ decision to adopt cloud computing in the UK. J Enterprise Inf Manag 2015 Oct 12;28(6):788-807 [FREE Full text] [CrossRef]
  44. Thiesse F, Staake T, Schmitt P, Fleisch E. The rise of the 'next‐generation bar code': an international RFID adoption study. Supply Chain Manag 2011 Aug 9;16(5):328-345 [FREE Full text] [CrossRef]
  45. Wang Y, Wang Y, Yang Y. Understanding the determinants of RFID adoption in the manufacturing industry. Technol Forecast Soc 2010 Jun;77(5):803-815 [FREE Full text] [CrossRef]
  46. Zhu K, Dong S, Xu S, Kraemer K. Innovation diffusion in global contexts: determinants of post-adoption digital transformation of European companies. Eur J Inform Syst 2017 Dec 19;15(6):601-616 [FREE Full text] [CrossRef]
  47. Zhu K, Kraemer K, Xu S. The process of innovation assimilation by firms in different countries: a technology diffusion perspective on e-business. Manag SCi 2006 Oct;52(10):1557-1576 [FREE Full text] [CrossRef]
  48. Gibbs J, Kraemer K. A cross-country investigation of the determinants of scope of e-commerce use: an institutional approach. Electron Mark 2004 Jun 1;14(2):124-137 [FREE Full text] [CrossRef]
  49. Xu S, Zhu K, Gibbs J. Global technology, local adoption: a cross-country investigation of internet adoption by companies in the United States and China. Electron Mark 2004 Apr 1;14(1):13-24 [FREE Full text] [CrossRef]
  50. Stewart K, Segars A. An empirical examination of the concern for information privacy instrument. Info Syst Res 2002 Mar;13(1):36-49 [FREE Full text] [CrossRef]
  51. Jeyaraj A, Rottman J, Lacity M. A review of the predictors, linkages, and biases in IT innovation adoption research. J Info Technol 2006 Mar;21(1):1-23 [FREE Full text] [CrossRef]
  52. Venkatesh V, Brown S, Maruping L, Bala H. Predicting different conceptualizations of system use: the competing roles of behavioral intention, facilitating conditions, and behavioral expectation. MIS Q 2008;32(3):483-502 [FREE Full text] [CrossRef]
  53. Sligo J, Gauld R, Roberts V, Villa L. A literature review for large-scale health information system project planning, implementation and evaluation. Int J Med Inform 2017 Jan;97:86-97. [CrossRef] [Medline]
  54. Goodhue D, Thompson R. Task-technology fit and individual performance. MIS Q 1995 Jun;19(2):213-236 [FREE Full text] [CrossRef]
  55. Lin A, Chen N. Cloud computing as an innovation: percepetion, attitude, and adoption. Int J Info Manag 2012 Dec;32(6):533-540 [FREE Full text] [CrossRef]
  56. Pagani M. Determinants of adoption of high speed data services in the business market: evidence for a combined technology acceptance model with task technology fit model. Inf Manag 2006 Oct;43(7):847-860 [FREE Full text] [CrossRef]
  57. Ministry OH. Guiding principles for grading nursing in general hospitals (for trial implementation). China Nurs Manag 2009;9(6):33-34.
  58. Chen H, Bates RA. Instrument Translation and Development Strategies for Cross-Cultural Studies, in Proceedings of the Academy of Human Resource Development 2005;693?700. 2019 Presented at: -; -; -.
  59. Moore G, Benbasat I. Development of an instrument to measure the perceptions of adopting an information technology innovation. Inf Syst Res 1991 Sep;2(3):192-222 [FREE Full text] [CrossRef]
  60. Kim D, Ferrin D, Rao H. A trust-based consumer decision-making model in electronic commerce: the role of trust, perceived risk, and their antecedents. Decis Support Syst 2008 Jan;44(2):544-564 [FREE Full text] [CrossRef]
  61. Bhattacherjee A, Hikmet N. Reconceptualizing organizational support and its effect on information technology usage: evidence from the health care sector. J Compt Info Syst 2008;48(4):69-76 [FREE Full text] [CrossRef]
  62. Yap C, Thong J, Raman K. Effect of government incentives on computerisation in small business. Eur J Inform Syst 2017 Dec 19;3(3):191-206 [FREE Full text] [CrossRef]
  63. Grandon E, Pearson J. Electronic commerce adoption: an empirical study of small and medium US businesses. Ing Manag 2004 Dec;42(1):197-216 [FREE Full text] [CrossRef]
  64. Premkumar G, Roberts M. Adoption of new information technologies in rural small businesses. Omega 1999 Aug;27(4):467-484 [FREE Full text] [CrossRef]
  65. Hair J, Black W, Babin B, Anderson R, Tatham R. Multivariate Data Analysis: With Readings. Edinburgh, UK: Pearson Education; 2010.
  66. Shareef M, Kumar V, Kumar U. Predicting mobile health adoption behaviour: a demand side perspective. J Cust Behav 2014 Oct 31;13(3):187-205 [FREE Full text] [CrossRef]
  67. Wu J, Wang S, Lin L. Mobile computing acceptance factors in the healthcare industry: a structural equation model. Int J Med Inform 2007 Jan;76(1):66-77. [CrossRef] [Medline]
  68. Villalba-Mora E, Casas I, Lupiañez-Villanueva F, Maghiros I. Adoption of health information technologies by physicians for clinical practice: the Andalusian case. Int J Med Inform 2015 Jul;84(7):477-485. [CrossRef] [Medline]
  69. Avgar AC, Litwin AS, Pronovost PJ. Drivers and barriers in health IT adoption: a proposed framework. Appl Clin Inform 2012;3(4):488-500 [FREE Full text] [CrossRef] [Medline]
  70. Vishwanath A, Scamurra SD. Barriers to the adoption of electronic health records: using concept mapping to develop a comprehensive empirical model. Health Informatics J 2007 Jun;13(2):119-134. [CrossRef] [Medline]


CRM: customer relationship management
e-business: electronic business
HIE: health information exchange
IT: information technology
mHealth: mobile health
RFID: radio-frequency identification
SME: small-to-medium–sized enterprise
TMS: top management support
TOE: technology-organization-environment


Edited by C Lovis; submitted 23.05.19; peer-reviewed by W Zhang, K Ahmed, Z Li; comments to author 07.08.19; revised version received 18.11.19; accepted 22.11.19; published 14.07.20

Copyright

©Boumediene Ramdani, Binheng Duan, Ilhem Berrou. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 14.07.2020.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.