This paper is in the following e-collection/theme issue:

Decision Support for Health Professionals (1161) Machine Learning (1432) Physician and Healthcare Worker Burnout (108)

Published on in Vol 8, No 5 (2020): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/16528, first published .
An App Developed for Detecting Nurse Burnouts Using the Convolutional Neural Networks in Microsoft Excel: Population-Based Questionnaire Study

An App Developed for Detecting Nurse Burnouts Using the Convolutional Neural Networks in Microsoft Excel: Population-Based Questionnaire Study

An App Developed for Detecting Nurse Burnouts Using the Convolutional Neural Networks in Microsoft Excel: Population-Based Questionnaire Study

Authors of this article:

Yi-Lien Lee1, 2 Author Orcid Image ;   Willy Chou3, 4 Author Orcid Image ;   Tsair-Wei Chien5 Author Orcid Image ;   Po-Hsin Chou6, 7 Author Orcid Image ;   Yu-Tsen Yeh8 Author Orcid Image ;   Huan-Fang Lee9 Author Orcid Image
Article Authors Cited by (26) Tweetations (3) Metrics

    Original Paper

    1Department of Medical Affairs, Chi Mei Medical Center, Tainan, Taiwan

    2Department of Information Management and Institute of Healthcare Information Management, National Chung Cheng University, Chayi, Taiwan

    3Department of Physical Medicine and Rehabilitation, Chiali Chi Mei Hospital, Chi Mei Medical Groups, Tainan, Taiwan

    4Department of Physical Medicine and Rehabilitation, Chung Shan Medical University, Taichun, Taiwan

    5Department of Medical Research, Chi Mei Medical Center, Chi Mei Medical Groups, Tainan, Taiwan

    6Department of Orthopedics and Traumatology, Taipei Veterans General Hospital, Taipei, Taiwan

    7School of Medicine, National Yang-Ming University, Taipei, Taiwan

    8Medical School, St George’s, University of London, London, United Kingdom

    9Department of Nursing, College of Medicine, National Cheng Kung University, Tainan, Taiwan

    *these authors contributed equally

    Corresponding Author:

    Huan-Fang Lee, PhD

    Department of Nursing

    College of Medicine

    National Cheng Kung University

    988 Chung Hwa Road

    Yung Kung District

    Tainan

    Taiwan

    Phone: 886 62812811

    Email: Eamonn0330@gmail.com


    Background: Burnout (BO), a critical syndrome particularly for nurses in health care settings, substantially affects their physical and psychological status, the institute’s well-being, and indirectly, patient outcomes. However, objectively classifying BO levels has not been defined and noticed in the literature.

    Objective: The aim of this study is to build a model using the convolutional neural network (CNN) to develop an app for automatic detection and classification of nurse BO using the Maslach Burnout Inventory–Human Services Survey (MBI-HSS) to help assess nurse BO at an earlier stage.

    Methods: We recruited 1002 nurses working in a medical center in Taiwan to complete the Chinese version of the 20-item MBI-HSS in August 2016. The k-mean and CNN were used as unsupervised and supervised learnings for dividing nurses into two classes (n=531 and n=471 of suspicious BO+ and BO−, respectively) and building a BO predictive model to estimate 38 parameters. Data were separated into training and testing sets in a proportion 70%:30%, and the former was used to predict the latter. We calculated the sensitivity, specificity, and receiver operating characteristic curve (area under the curve) across studies for comparison. An app predicting respondent BO was developed involving the model’s 38 estimated parameters for a website assessment.

    Results: We observed that (1) the 20-item model yields a higher accuracy rate (0.95) with an area under the curve of 0.97 (95% CI 0.94-0.95) based on the 1002 cases, (2) the scheme named matching personal response to adapt for the correct classification in model drives the prior model’s predictive accuracy at 100%, (3) the 700-case training set with 0.96 accuracy predicts the 302-case testing set reaching an accuracy of 0.91, and (4) an available MBI-HSS app for nurses predicting BO was successfully developed and demonstrated in this study.

    Conclusions: The 20-item model with the 38 parameters estimated by using CNN for improving the accuracy of nurse BO has been particularly demonstrated in Excel (Microsoft Corp). An app developed for helping nurses to self-assess job BO at an early stage is required for application in the future.

    JMIR Med Inform 2020;8(5):e16528

    doi:10.2196/16528

    Keywords

    nurse burnout; MBI-HSS Chinese version; receiver operating characteristic curve; convolutional neural network; Lz person fit statistic 


    Burnout in the Workplace

    Burnout (BO) is a critical syndrome and problem in high-tech service-oriented societies, particularly for nurses in health care settings [1-4]. Many studies [5-11] reported that BO influences an employee’s physical and psychological status [5-7], the organizational well-being [8-11], and patient quality-of-care outcomes [6,10].

    One of the most popular BO inventories is the Maslach Burnout Inventory–Human Services Survey (MBI-HSS) [12,13]. More than 1898 articles were found by searching the keywords “Maslach” and “burnout” on September 23, 2019. BO is defined by Maslach [12,13] as a syndrome of emotional exhaustion, reduced personal accomplishment (PA), and depersonalization that frequently occurs in individuals who work in people-related jobs, such health care and educational.

    Maslach Burnout Inventory–Human Services Survey

    The MBI-HSS [13] has been widely applied to measure individual BO in numerous workplaces [4,11,14-16]. The original MBI-HSS is a 22-item inventory with a 7-point scale (from never=0 to every day=6) to measure BO for workers in a recent week [13]. The three BO subscales comprise 9 items for emotional exhaustion, 8 items for personal accomplishment, and 5 items for depersonalization. Despite the survey being popularly used in social science, the cutting point for determining BO substantially differs between cultures and health care settings [15,17-21]. Accordingly, Maslach et al [22] suggested that BO levels (low, moderate, and high) had different cutting points in different countries and areas. Schaufeli and Van Dierendonck [23] suggested having common cutting points to compare BO levels among countries and areas.

    Maslach and Jackson [13] suggested that the cutting points be set at 54 for emotional exhaustion, 48 for personal accomplishment, and 30 for depersonalization using subscale scores for measurement. Schaufeli and Van Dierendonck [23] were critical of the fact that the scheme for determining BO levels was arbitrary based on the three groups that contained an equal number of sample sizes [24]. Although Maslach and Jackson [13] also suggested having valid criteria that can be used for classifying BO levels, no such reasonable and viable scheme has been accepted by practitioners in the past.

    Convolutional Neural Network

    Convolutional neural network (CNN) has had the greatest impact within the field of health informatics [25]. Its architecture can be described as an interleaved set of feedforward layers implementing convolutional filters followed by reduction, rectification, or pooling layers [26-28]. For each layer, the CNN creates a high-level abstract feature. Whether the CNN, a famous deep learning method, can improve the prediction accuracy (up to 7.14%) [28] on nurse BO classification is worthy of study.

    Online Classification Using Smartphones is Required

    As with all forms of web-based technology, advances in mobile health communication technology are rapidly increasing [29]. Until now, there has been no app for smartphones to classify nurse BO levels. If the CNN BO model’s parameters have been estimated by the CNN algorithm, the classification of nurse BO by responding to the MBI-HSS can alert individual nurses more accurately and warn them to alleviate their mental strain before it becomes a serious BO problem.

    Study Aims

    The aims of our study are to (1) estimate the model’s parameters using CNN based on nurse responses to the MBI-HSS and (2) design an app for smartphones based on a website assessment of nurse BO.


    Data Source

    Study Sample and Demographic Data

    If the confidence level and intervals were set at 0.05 and ±5% and applied to the population of 1850 registered nurses in a hospital, 318 participants are required for the sample size [30]. We estimated the rate of refusal to respond will reach 40%. The minimum number for the study sample size will be 540 (318/[1–0.4]).

    We delivered 40 copies each of the MBI-HSS BO survey to 32 nursing units. A sample of 1255 registered nurses with at least 1 month experience in the Chi Mei Medical Center (Taiwan) was randomly selected to complete the Chinese version of the 20-item MBI-HSS [3] in August 2016. A total of 1002 participants were eligible, for a return rate of 79.9%.

    Featured Variables

    Featured variables consist of the 20 items (called the 20-item model in which the response in the subscale of reduced personal accomplishment has been reversed to be the higher score denoting the more serious BO problem) on the classification of nurse BO levels (ie, suspicious BO+ and BO−). The 1002 participants were split into training and testing sets in a proportion (70%:30%), and the former was used to predict the latter. The data are shown in Multimedia Appendix 1. This study was approved and monitored by the Chi Mei Medical Center institutional review board (10704-003). All hospital and study participant identifiers were stripped.

    Unsupervised and Supervised Learnings

    Unsupervised learning indicates agnostic aggregation of unlabeled data sets yielding groups or clusters of entities with shared similarities that may be unknown prior to the analysis step [31,32] (eg, clustering dimensionality reduction using principle component analysis or k-mean clustering). The k-mean clustering aims to partition n observations into k clusters, in which each observation belongs to the cluster with the nearest mean [33]. In contrast, supervised learning employs labeled training data sets (labeled/supervised by subject experts or by the objective k-mean clustering) to yield a qualitative or quantitative output [31,34].

    In this study, the k-mean was used as unsupervised learning for clustering participants into two classes (n=531 and n=471 for suspicious BO+ and BO−, respectively). CNN was applied as supervised learning to build a BO prediction model for estimating the 38 parameters.

    Convolutional Neural Network Applied in This Study

    CNN is a variant of the standard multilayer perceptron, especially used for pattern recognition compared with conventional approaches [35] due to its capability in reducing the dimension of data, extracting the feature sequentially, and classifying one structure of the network [36]. The basic CNN model was inspired in 1962 from the visual cortex proposed by Hubel and Wiesel [35]. For simplifying the CNN concept and process, we present it in Figure 1. Detailed information on interpretation is provided in Multimedia Appendix 2.

    ‎
    Figure 1. Interpretation of the convolutional neural network algorithm.
    View this figure

    Tasks for Performing Convolutional Neural Network

    Task 1: Comparison of Prediction Accuracies in Two Modes

    Two sets of featured variables (ie, 20 with the traditional accurate rate and 100% rate) on 1002 cases were mirrored to compare the prediction accuracies (eg, sensitivity, specificity, and receiver operating characteristic (ROC) curve [area under the curve, AUC]) using the CNN algorithm.

    In contrast to the traditionally predictive method, we use the known responses and their corresponding labels (ie, suspicious BO+ or BO−) to build a model for predicting the unknown label of the specific responses. The reason for reaching a 100% accuracy rate on the known responses and their corresponding labels in the training set is to avoid letting the CNN fail in the classification of the known responses in the future. A scheme named matching personal response scheme to adapt for the correct classification in the model (MPRSA) is designed for driving the model’s accuracy toward 100%. The way we applied the MPRSA is presented for achieving this 100% goal if the same response string is encountered in the future: the MPRSA is regarding the original responses (eg, the 20-item string coded as 9223372036854775807) that are linked to the correct label in the validation or testing set through which all cases in the training set would reach a 100% accuracy rate if the cases are present in the testing set.

    Task 2: Validation Compared With the Training and Testing Sets

    The 1002 cases were split into training and testing sets in a proportion of 70%:30%, and the former was used to predict the latter. The accuracy rates in these two sets were compared.

    Task 3: App Detecting Burnout for a Web-Based Assessment

    A 20-item self-assessment app using participant mobile phones was designed to predict nurse BO using the CNN algorithm and the model parameters [37]. The resulting classification appears on smartphones. The visual representation with binary (BO– and BO+) category probabilities is shown on a dashboard using Google Maps to display.

    Statistical Tools and Data Analysis

    MedCalc 9.5.0.0 for Windows (MedCalc Software) was used to calculate the sensitivity, specificity, and corresponding AUC using logistic regression when the observed labels (ie, 0 for BO– and 1 for BO+) and the predicted probabilities (ie, the continuous variable in step 3 calculated by the sigmoid function in the output layer in Figure 1) were applied. A visual representation displaying the classification effect is plotted using two curves (ie, one from the left-bottom to the right-top corner denotes the success [BO+] feature and another from the left-top corner to the right-bottom side as the failure attribute). The study flowchart and the CNN modeling process are shown in Figure 2 and Multimedia Appendix 2, respectively.

    ‎
    Figure 2. Study flowchart. CNN: convolutional neural network; MPRSA: matching personal response scheme to adapt for the correct classification in the model.
    View this figure

    Demographic Data of Participants

    The demographic data of the nurses are shown in Table 1. We can see that females accounted for 93.1% (933/1002) of the participants. Most participants had a bachelor’s (university) degree (892/1002, 89.0%). The single accounted for 59.5% (596/1002), and the married (399/1002, 39.8%). Among the nurses, 37.3% (37/1002) had work experience outside the study hospital, while 62.5% (627/1002) had none.

    The highest in nurse hierarchy is N (132/1002, 13.2%), followed by N1 (134/1002, 13.4%), N2 (272/1002, 27.1%), N3 (248/1002, 24.8%), and N4 (215/1002, 21.5%). The top two job titles are nurse (797/1002, 79.5%) and leader (149/1002, 14.9%).

    The average age for the sample is 32.6 (SD 7.2) years, ranging from 23 to 56. The average work experience in other hospitals reaches 15.1 (SD 28.5) months.

    The workload in terms of the number of patients cared for in a week by each nurse averages 11 (SD 19.1). The mean for non-care affairs in a week reaches 4 hours (SD 5.8). The mean of nursing care is 9 (SD 2.7) hours per week. The average number of a patient cared for is 9 (SD 12.1).

    Table 1. Demographic data of the study sample.
    Variable and typeValue
    Gender, n (%)

    		Male69 (6.9)

    		Female933 (93.1)
    Education, n (%)

    		Less than university46 (4.6)

    		University892 (89.0)

    		Graduate school64 (6.4)
    Marital status, n (%)

    		Single596 (59.5)

    		Married399 (39.8)

    		Divorced7 (0.7)
    Work tenure, n (%)

    		Without627 (62.6)

    		With375 (37.4)
    Nurse hierarchy, n (%)

    		N (<1 year experience)133 (13.3)

    		N1 (Fundamentals of Nursing)134 (13.4)

    		N2 (Critical Care in Nursing)272 (27.1)

    		N3 (Holistic Care and Teaching)248 (24.8)

    		N4 (Specialist Nursing and Research)215 (21.5)
    Job title, n (%)

    		Nurse798 (79.6)

    		Leader147 (14.7)

    		Assistant head nurse30 (3.0)

    		Head nurse27 (2.7)
    Age, mean (SD), range32.6 (7.2), 23-56
    Work experience outside hospital (month), mean (SD), range15.1 (28.5), 0-180
    Average hours spent in non-care affairs per week, mean (SD), range3.9 (5.8), 0-60
    Average weekly hours spent in nursing care, mean (SD), range9.2 (2.9), 1.5-70
    Average daily patient care, mean (SD), range9.5 (12.1), 0-120

    Unsupervised Learnings Using the K-Mean Clustering

    A visual representation displaying the classification effect is plotted using the box plot (Figure 3). We can see a smaller number of cases with suspicious BO– having a higher total score, and a smaller number of cases are misclassified as BO+ (12.1%) and BO− (9.6%). In contrast, the sensitivity and specificity are 90.4% and 87.9%, respectively. The cutting point is set at 43 with an AUC 0.96 (bottom, Figure 3) if the unsupervised learning approach is applied.

    ‎
    Figure 3. Two study groups divided by the k-mean algorithm (A) and receiver operating characteristic curve (B).
    View this figure

    Tasks to Compare the Accuracy Between Modes

    Comparison of Prediction Accuracies in Two Modes

    The 20-item model yields a higher accuracy rate (0.95) with an AUC 0.98 (95% CI 0.97-1.00) higher than that of the 20-item model with an accuracy of 0.95 and an AUC 0.97 (95% CI 0.96-0.99) based on the 1002 cases.

    The MPRSA applied to the bottom pattern in Table 2 drives the model’s accuracy at 100%.

    Validation Compared With the Training and Testing Sets

    The 700-case training set with 0.96 accuracies predicts the 302-case testing set reaching an accuracy of 0.91 (Table 3).

    Table 2. Three scenarios applied to convolutional neural network for the prediction of nurse burnout (n=1002).
    SampleTrue condition

    		BO+aBO–bBO+/row #BO–/row #
    Scenario A (only 20 items)




    		Positive507260.950.05

    		Negative244450.050.95
    Scenario B (Scenario A and MPRSAc) training




    		Positive53101.000

    		Negative047101.00

    aBO+: suspicious for burnout.

    bBO–: not suspicious for burnout.

    cMPRSA: matching personal response scheme to adapt for the correct classification.

    Table 3. Training and testing effects.
    SampleTrue condition

    		BO+aBO–bBO+/row #BO–/row #
    Scenario A (20 items) training, n=700




    		Positive362150.960.04

    		Negative103130.030.97
    Scenario B (20 items) testing, n=302




    		Positive147160.900.10

    		Negative111280.080.92

    aBO+: suspicious for burnout.

    bBO–: not suspicious for burnout.

    App Detecting Burnout for a Web-Based Assessment

    An MBI-HSS app for nurses predicting BO was developed (Figure 4). Interested readers are invited to scan the QR code to practice the MBI-HSS app on their own. It is worth noting that all 38 model parameters are embedded in the 20-item CNN model for classification of either suspicious BO+ or BO− once all 20 items have been responded to.

    One resulting example is present in Figure 5, from which we can see that the BO– with a high probability (0.99) is shown on the curve of the failure from the left-top to the right-bottom corner. The sum of both probabilities (ie, BO+ and BO–) equals 1.0. The odds can be computed by the formula (p/[1–p]=0.01/0.99=0.01), indicating the nurse with an extremely low probability or tendency toward BO+.

    ‎
    Figure 4. Screenshot of the mobile phone app.
    View this figure
    ‎
    Figure 5. The result of assessing nurse burnout.
    View this figure

    Principal Findings

    We observed that (1) the 20-item model yields a higher accuracy rate (0.95; AUC 0.97, 95% CI 0.94-0.95), (2) the MPRSA drives the model’s prior accuracy at 100%, (3) the 700-case training set with 0.96 accuracy predicts the 302-case testing set reaching an accuracy of 0.91, and (4) the MBI-HSS app for nurses predicting BO has been developed and demonstrated.

    The MBI-HSS is the most widely used tool for measuring BO in the world [11,14-16]. More than 1898 articles were found by searching the keywords “Maslach” and “burnout” on September 23, 2019. However, none provided an acceptable scheme to classify BO levels (ie, BO+ and BO− or low, moderate, and high).

    Maslach and Jackson [13] provided a cutting point scheme (ie, 54 for emotional exhaustion, 48 for personal accomplishment, and 30 for depersonalization; around 40=132/3*20/22) approximately equal to 43 (based on 20 items using subscale total scores; see Figure 3) in this study. Although Schaufeli and Van Dierendonck [23] doubted that the cutting points proposed by Maslach and Jackson [13] were arbitrary only on the assumption of an equal sample size across the levels (ie, high, moderate, and low), our cutting point at 43 is derived through the k-mean clustering.

    However, no matter which cutting point scheme is applied, that of Maslach and Jackson [13] or this study (eg, in Figure 3), misclassifications must exist due to their type I (α) and II (1–β) errors. In contrast, the CNN predictive model combined with the MPRSA mentioned in Methods regarding task 1 (100% accuracy rate is required) can minimize the type I and II errors toward zero (eg, Table 2), which is one of the features of this study.

    Implications and Future Work

    CNN can improve prediction accuracy (up to 7.14%) [28]. In this study, sensitivity and specificity have been improved. So far, we have not seen anyone using the CNN approach to predict nurse BO in the literature, which is a breakthrough, and the first feature of this study.

    Over 708 articles have been found using the keyword “convolutional neural network” (Title) searched in PubMed Central on September 23, 2019. None used Microsoft Excel to perform the CNN. The interpretations for the CNN concept and process or the parameter estimations are shown in Figure 1, Multimedia Appendix 2 and 3, and in the app [38], which is the second feature of this study.

    Using Microsoft Excel to perform CNN is the third feature of this study (Multimedia Appendix 1), which was rarely seen applicable in the literature.

    Because the principle for concerning more with the vital few and less with the trivial numerous is emphasized in the quality control process, we propose the MPRSA as the fourth feature. We incorporated the original responses into the model to let the label be correctly classified by the CNN, through which all cases with a false prediction in the training set would be adjusted as a true prediction, reaching a 100% accuracy rate if the cases reoccur in the testing set.

    Furthermore, the curves of category probabilities based on the Rasch rating scale model [39] are shown in Figure 4. The binary categories (eg, success and failure on an assessment in the psychometric field) have been applied in health-related outcomes [40-44]. However, none provided the animation-type dashboard showing on Google Maps, as we did in Figure 4.

    Strengths

    It is easy to set up the nurse BO online assessment if the designer uploads relevant and appropriate audio and visual files to the corresponding questions of the database. We applied the CNN algorithm along with the model’s parameters to design the routine on an app that is used to detect BO risk for nurses in hospitals (Figure 4), which has never been seen before for the MBI-HSS [13] implemented on mobile phones.

    As with all forms of web-based technology, advances in health communication technology are rapidly emerging [29]. Mobile online BO assessment is promising and worth considering in many fields of health assessment. An online BO assessment (Figure 4) can be applied to inform examinees quickly about when and whether they should take actions or follow up to see a psychiatrist and how to improve their behaviors and attitudes given that their lifestyle is not changed [4]. The online BO assessment is promising, and it is worth using for promoting nurses’ health literacy by using the animation-type assessment on smartphones. Interested readers are recommended to scan the QR codes on Figure 4, one for the app and another for the MP4, and see the details about responding to questions and the real experience on answering the 20-item MBI-HSS for a website assessment.

    The CNN module on Microsoft Excel is unique and innovative (Multimedia Appendix 1). Users who are not familiar with the CNN software (eg, Python) can apply our Excel Visual Basic for Applications module to conduct CNN-related research in the future. The module is not limited to the binary classification. The multiclassification module can be done by adding the layers on CNN. That is, two categories require two input layers and two pooling layers. Similarly, three categories need three input layers and three pooling layers (Figure 1 and Multimedia Appendix 1 and 2). Any other types of self-assessment, such as work bullying, depression, and dengue fever, can apply the CNN model to predict and classify the levels of harmfulness and disease in the future.

    Limitations and Suggestions

    Our study has some limitations. First, although the psychometric properties of the 20-item MBI-HSS have been validated for measuring nurse BO in Taiwan [3] after removing item 14 (I feel I am working too hard on my job) and item 22 (I feel patients blame me for some of their problems), there is no evidence that supports that the 20-item MBI-HSS is suitable for nurses in other regions. We recommend additional studies using their own k-mean algorithm and the CNN model to estimate the parameters and see the difference (eg, the cutting point at 43 in Figure 3).

    Second, we have not discussed any improvement in predictive accuracy. For instance, whether other featured variables (eg, the mean, SD, and Lz index [44,45]) applied to the CNN model can increase the accurate rate is worthy of further study. Future studies are needed to look for other variables that can improve the power of the model prediction.

    Third, the study was based on previously published [3] research using the 20-item MBI-HSS. All of the data were sampled from similar health care settings. If any environment or condition is changed (eg, other professionals or workplaces), the result (eg, the model’s parameters) must be different from this study.

    Fourth, the MBI-HSS is a three-dimensional construct. Usually, the item difficulties should be first calibrated by using the Rasch ConQuest software [46]. The CNN model [47] can ignore the issue of dimensionality and gain a favorable prediction effect that should be verified and ensured in the future.

    Finally, the study sample was taken from Taiwanese data in a nurse survey. The model parameters estimated for the MBI-HSS Chinese version are only suitable for the Chinese (particularly for Taiwanese) society in health care settings. Generalizing these BO assessment findings (eg, the cutting point at around 43; see Figure 3) might be somewhat limited and constrained because the sample merely consisted of nurses working for inpatients. Additional studies are needed to reexamine whether the psychometric properties of the BO assessment are similar to that of other worksites in (or out of) a hospital.

    Conclusion

    We illustrate features and contributions in this study: (1) CNN performed in Microsoft Excel, (2) MPRSA applied to increase the model’s prior prediction accuracy, (3) an online app demonstrated to display results using a visual dashboard on Google Maps, and (4) the category probability curves based on Rasch rating scale model first observed in the CNN prediction model. The novelty of the app with the CNN algorithm improves the predictive accuracy of nurse BO. It is expected to help nurses self-assess job BO at an early stage in the future.

    Authors' Contributions

    YLL conceived and designed the study, WC and PHC performed the statistical analyses, and YTY was in charge of recruiting study participants. TWC helped design the study, collected information, and interpreted data. HFL monitored the research. All authors read and approved the final article.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Study dataset.

    XLSX File (Microsoft Excel File), 132 KB
    ‎
    Multimedia Appendix 2

    Convolutional neural network to interpret Figure 1.

    DOCX File , 81 KB
    ‎
    Multimedia Appendix 3

    Mp4 for convolutional neural network performed in Excel.

    TXT File , 0 KB
    1. Abushaikha L, Saca Hazboun H. Job satisfaction and burnout among Palestinian nurses. East Mediterr Health J 2009 Jan 01;15(1):190-197. [CrossRef]
    2. Demir A, Ulusoy M, Ulusoy MF. Investigation of factors influencing burnout levels in the professional and private lives of nurses. International Journal of Nursing Studies 2003 Nov;40(8):807-827. [CrossRef]
    3. Lee H, Chien T, Yen M. Examining factor structure of Maslach burnout inventory among nurses in Taiwan. J Nurs Manag 2013;21:648-656. [CrossRef] [Medline]
    4. Lee H. Determining cutting points of the Maslach Burnout Inventory for nurses to measure their level of burnout online. History Res 2017;5(1):1-8. [CrossRef]
    5. Hsu H, Chen S, Yu H, Lou J. Job stress, achievement motivation and occupational burnout among male nurses. J Adv Nurs 2010 Jul;66(7):1592-1601. [CrossRef] [Medline]
    6. Spence Laschinger HK, Leiter MP. The impact of nursing work environments on patient safety outcomes: the mediating role of burnout/engagement. J Nurs Adm 2006 May;36(5):259-267. [CrossRef] [Medline]
    7. Trinkoff A, Geiger-Brown J, Brady B, Lipscomb J, Muntaner C. How long and how much are nurses now working? Am J Nurs 2006 Apr;106(4):60-71. [CrossRef] [Medline]
    8. Alacacioglu A, Yavuzsen T, Dirioz M, Oztop I, Yilmaz U. Burnout in nurses and physicians working at an oncology department. Psychooncology 2009 May;18(5):543-548. [CrossRef] [Medline]
    9. Garrett C. The effect of nurse staffing patterns on medical errors and nurse burnout. AORN J 2008 Jun;87(6):1191-1204. [CrossRef] [Medline]
    10. Halbesleben JRB, Wakefield BJ, Wakefield DS, Cooper LB. Nurse burnout and patient safety outcomes: nurse safety perception versus reporting behavior. West J Nurs Res 2008 Aug;30(5):560-577. [CrossRef] [Medline]
    11. Spence Laschinger HK, Leiter M, Day A, Gilin D. Workplace empowerment, incivility, and burnout: impact on staff nurse recruitment and retention outcomes. J Nurs Manag 2009 Apr;17(3):302-311. [CrossRef] [Medline]
    12. Maslach C. Burnout: The Cost of Caring. Englewood Cliffs: Prentice-Hall; 1982.
    13. Maslach C, Jackson S. Maslach Burnout Inventory Manual, 2nd Edition. Palo Alto: Consulting Psychologists Press; 1986.
    14. Li XM, Liu YJ. [Job stressors and burnout among staff nurses]. Chin J Nurs 2000;35(11):645.
    15. Lin F, St John W, McVeigh C. Burnout among hospital nurses in China. J Nurs Manag 2009 Apr;17(3):294-301. [CrossRef] [Medline]
    16. Tourangeau A, Cummings G, Cranley L, Ferron E, Harvey S. Determinants of hospital nurse intention to remain employed: broadening our understanding. J Adv Nurs 2010 Jan;66(1):22-32 [FREE Full text] [CrossRef] [Medline]
    17. Beckstead JW. Confirmatory factor analysis of the Maslach Burnout Inventory among Florida nurses. Int J Nurs Stud 2002 Nov;39(8):785-792. [CrossRef] [Medline]
    18. Kanste O, Miettunen J, Kyngäs H. Factor structure of the Maslach Burnout Inventory among Finnish nursing staff. Nurs Health Sci 2006 Dec;8(4):201-207. [CrossRef] [Medline]
    19. Poghosyan L, Aiken LH, Sloane DM. Factor structure of the Maslach burnout inventory: an analysis of data from large scale cross-sectional surveys of nurses from eight countries. Int J Nurs Stud 2009 Jul;46(7):894-902 [FREE Full text] [CrossRef] [Medline]
    20. Vanheule S, Rosseel Y, Vlerick P. The factorial validity and measurement invariance of the Maslach Burnout Inventory for human services. Stress Health 2007 Apr;23(2):87-91. [CrossRef]
    21. Worley JA, Vassar M, Wheeler DL, Barnes LLB. Factor structure of scores from the Maslach Burnout Inventory. Educat Psychol Measure 2008 Feb 05;68(5):797-823. [CrossRef]
    22. Maslach C, Schaufeli WB, Leiter MP. Job burnout. Annu Rev Psychol 2001;52:397-422. [CrossRef] [Medline]
    23. Schaufeli WB, Van Dierendonck D. A cautionary note about the cross-national and clinical validity of cut-off points for the Maslach Burnout Inventory. Psychol Rep 1995 Jun;76(3 Pt 2):1083-1090. [CrossRef] [Medline]
    24. Golembiewski R, Deckard G, Roundtree B. The stability of burnout assignment: measurement properties of the phase model. J Health Hum Serv Admin 1989;12:63-78.
    25. Ravi D, Wong C, Deligianni F, Berthelot M, Andreu-Perez J, Lo B, et al. Deep learning for health informatics. IEEE J Biomed Health Inform 2017 Jan;21(1):4-21. [CrossRef] [Medline]
    26. Tobore I, Li J, Yuhang L, Al-Handarish Y, Kandwal A, Nie Z, et al. Deep learning intervention for health care challenges: some biomedical domain considerations. JMIR Mhealth Uhealth 2019 Aug 02;7(8):e11966 [FREE Full text] [CrossRef] [Medline]
    27. Kwon S, Hong J, Choi E, Lee E, Hostallero DE, Kang WJ, et al. Deep learning approaches to detect atrial fibrillation using photoplethysmographic signals: algorithms development study. JMIR Mhealth Uhealth 2019 Jun 06;7(6):e12770 [FREE Full text] [CrossRef] [Medline]
    28. Sathyanarayana A, Joty S, Fernandez-Luque L, Ofli F, Srivastava J, Elmagarmid A, et al. Sleep quality prediction from wearable data using deep learning. JMIR Mhealth Uhealth 2016 Nov 04;4(4):e125 [FREE Full text] [CrossRef] [Medline]
    29. Mitchell SJ, Godoy L, Shabazz K, Horn IB. Internet and mobile technology use among urban African American parents: survey study of a clinical population. J Med Internet Res 2014 Jan 13;16(1):e9 [FREE Full text] [CrossRef] [Medline]
    30. Survey System. Sample Size Calculator   URL: https://www.surveysystem.com/sscalc.htm [accessed 2020-01-14]
    31. Rashidi HH, Tran NK, Betts EV, Howell LP, Green R. Artificial intelligence and machine learning in pathology: the present landscape of supervised methods. Acad Pathol 2019;6:2374289519873088 [FREE Full text] [CrossRef] [Medline]
    32. Buehler L, Rashidi H. Bioinformatics Basics, Application in Biological Science and Medicine, 2nd Edition. Philadelphia: Taylor and Francis Group; 2005.
    33. Chen C, Luo J, Parker K. Image segmentation via adaptive K-mean clustering and knowledge-based morphological operations with biomedical applications. IEEE Trans Image Process 1998;7(12):1673-1683. [CrossRef] [Medline]
    34. Caruana R, Niculescu-Mizil A. An empirical comparison of supervised learning algorithms. 2006 Presented at: Proceedings of the 23rd International Conference on Machine Learning; 2006; Pittsburgh p. 161-168   URL: https://dl.acm.org/doi/10.1145/1143844.1143865 [CrossRef]
    35. Guzmán MG, Kourí G. Dengue: an update. Lancet Infect Dis 2002 Jan;2(1):33-42. [CrossRef] [Medline]
    36. Bengio Y. Learning deep architectures for AI. Found Trends Mach Learn 2009;2(1):1-127 [FREE Full text] [CrossRef]
    37. Chien T, Lin W. Simulation study of activities of daily living functions using online computerized adaptive testing. BMC Med Inform Decis Mak 2016 Oct 10;16(1):130 [FREE Full text] [CrossRef] [Medline]
    38. iHelp.   URL: http://www.healthup.org.tw/irs/irsin_e.asp?type1=87 [accessed 2020-01-22]
    39. Andrich D. A rating formulation for ordered response categories. Psychometrika 1978 Dec;43(4):561-573. [CrossRef]
    40. Lee Y, Lin K, Chien T. Application of a multidimensional computerized adaptive test for a Clinical Dementia Rating Scale through computer-aided techniques. Ann Gen Psychiatry 2019;18:5 [FREE Full text] [CrossRef] [Medline]
    41. Ma S, Wang H, Chien T. A new technique to measure online bullying: online computerized adaptive testing. Ann Gen Psychiatry 2017;16:26 [FREE Full text] [CrossRef] [Medline]
    42. Ma S, Chien T, Wang H, Li Y, Yui M. Applying computerized adaptive testing to the Negative Acts Questionnaire-Revised: Rasch analysis of workplace bullying. J Med Internet Res 2014 Feb 17;16(2):e50 [FREE Full text] [CrossRef] [Medline]
    43. Chien T, Lin W. Improving inpatient surveys: web-based computer adaptive testing accessed via mobile phone QR codes. JMIR Med Inform 2016 Mar 02;4(1):e8 [FREE Full text] [CrossRef] [Medline]
    44. Hulin C, Drasgow F, Parsons C. Item Response Theory: Applications to Psychological Measurement. Homewood: Dow & Jones Irwin; 1983.
    45. Linacre J. An all-purpose person fit statistic? Rasch Measure Transact 1997;11(3):582-583 [FREE Full text]
    46. Wu M, Adams R, Wilson M. Acer ConQuest. Melbourne: Australian Council for Educational Research Press; 1998.
    47. Saha S. A comprehensive guide to convolutional neural networks—the eli5 way. 2018 Dec 15.   URL: https:/​/towardsdatascience.​com/​a-comprehensive-guide-to-convolutional-neural-networks-the-eli5-way-3bd2b1164a53 [accessed 2020-01-14]

    ‎
    AUC: area under the curve
    BO: burnout
    CNN: convolutional neural network
    MBI-HSS: Maslach Burnout Inventory–Human Services Survey
    MPRSA: matching personal response scheme to adapt for the correct classification
    PA: personal accomplishment
    ROC: receiver operation characteristic

    Edited by C Lovis; submitted 07.10.19; peer-reviewed by S Probst, S Chen; comments to author 15.12.19; revised version received 15.12.19; accepted 31.12.19; published 07.05.20

    Copyright

    ©Yi-Lien Lee, Willy Chou, Tsair-Wei Chien, Po-Hsin Chou, Yu-Tsen Yeh, Huan-Fang Lee. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 07.05.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.