This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on http://medinform.jmir.org/, as well as this copyright and license information must be included.
Burnout (BO), a critical syndrome particularly for nurses in health care settings, substantially affects their physical and psychological status, the institute’s well-being, and indirectly, patient outcomes. However, objectively classifying BO levels has not been defined and noticed in the literature.
The aim of this study is to build a model using the convolutional neural network (CNN) to develop an app for automatic detection and classification of nurse BO using the Maslach Burnout Inventory–Human Services Survey (MBI-HSS) to help assess nurse BO at an earlier stage.
We recruited 1002 nurses working in a medical center in Taiwan to complete the Chinese version of the 20-item MBI-HSS in August 2016. The k-mean and CNN were used as unsupervised and supervised learnings for dividing nurses into two classes (n=531 and n=471 of suspicious BO+ and BO−, respectively) and building a BO predictive model to estimate 38 parameters. Data were separated into training and testing sets in a proportion 70%:30%, and the former was used to predict the latter. We calculated the sensitivity, specificity, and receiver operating characteristic curve (area under the curve) across studies for comparison. An app predicting respondent BO was developed involving the model’s 38 estimated parameters for a website assessment.
We observed that (1) the 20-item model yields a higher accuracy rate (0.95) with an area under the curve of 0.97 (95% CI 0.94-0.95) based on the 1002 cases, (2) the scheme named matching personal response to adapt for the correct classification in model drives the prior model’s predictive accuracy at 100%, (3) the 700-case training set with 0.96 accuracy predicts the 302-case testing set reaching an accuracy of 0.91, and (4) an available MBI-HSS app for nurses predicting BO was successfully developed and demonstrated in this study.
The 20-item model with the 38 parameters estimated by using CNN for improving the accuracy of nurse BO has been particularly demonstrated in Excel (Microsoft Corp). An app developed for helping nurses to self-assess job BO at an early stage is required for application in the future.
Burnout (BO) is a critical syndrome and problem in high-tech service-oriented societies, particularly for nurses in health care settings [
One of the most popular BO inventories is the Maslach Burnout Inventory–Human Services Survey (MBI-HSS) [
The MBI-HSS [
Maslach and Jackson [
Convolutional neural network (CNN) has had the greatest impact within the field of health informatics [
As with all forms of web-based technology, advances in mobile health communication technology are rapidly increasing [
The aims of our study are to (1) estimate the model’s parameters using CNN based on nurse responses to the MBI-HSS and (2) design an app for smartphones based on a website assessment of nurse BO.
If the confidence level and intervals were set at 0.05 and ±5% and applied to the population of 1850 registered nurses in a hospital, 318 participants are required for the sample size [
We delivered 40 copies each of the MBI-HSS BO survey to 32 nursing units. A sample of 1255 registered nurses with at least 1 month experience in the Chi Mei Medical Center (Taiwan) was randomly selected to complete the Chinese version of the 20-item MBI-HSS [
Featured variables consist of the 20 items (called the 20-item model in which the response in the subscale of reduced personal accomplishment has been reversed to be the higher score denoting the more serious BO problem) on the classification of nurse BO levels (ie, suspicious BO+ and BO−). The 1002 participants were split into training and testing sets in a proportion (70%:30%), and the former was used to predict the latter. The data are shown in
Unsupervised learning indicates agnostic aggregation of unlabeled data sets yielding groups or clusters of entities with shared similarities that may be unknown prior to the analysis step [
In this study, the k-mean was used as unsupervised learning for clustering participants into two classes (n=531 and n=471 for suspicious BO+ and BO−, respectively). CNN was applied as supervised learning to build a BO prediction model for estimating the 38 parameters.
CNN is a variant of the standard multilayer perceptron, especially used for pattern recognition compared with conventional approaches [
Interpretation of the convolutional neural network algorithm.
Two sets of featured variables (ie, 20 with the traditional accurate rate and 100% rate) on 1002 cases were mirrored to compare the prediction accuracies (eg, sensitivity, specificity, and receiver operating characteristic (ROC) curve [area under the curve, AUC]) using the CNN algorithm.
In contrast to the traditionally predictive method, we use the known responses and their corresponding labels (ie, suspicious BO+ or BO−) to build a model for predicting the unknown label of the specific responses. The reason for reaching a 100% accuracy rate on the known responses and their corresponding labels in the training set is to avoid letting the CNN fail in the classification of the known responses in the future. A scheme named matching personal response scheme to adapt for the correct classification in the model (MPRSA) is designed for driving the model’s accuracy toward 100%. The way we applied the MPRSA is presented for achieving this 100% goal if the same response string is encountered in the future: the MPRSA is regarding the original responses (eg, the 20-item string coded as 9223372036854775807) that are linked to the correct label in the validation or testing set through which all cases in the training set would reach a 100% accuracy rate if the cases are present in the testing set.
The 1002 cases were split into training and testing sets in a proportion of 70%:30%, and the former was used to predict the latter. The accuracy rates in these two sets were compared.
A 20-item self-assessment app using participant mobile phones was designed to predict nurse BO using the CNN algorithm and the model parameters [
MedCalc 9.5.0.0 for Windows (MedCalc Software) was used to calculate the sensitivity, specificity, and corresponding AUC using logistic regression when the observed labels (ie, 0 for BO– and 1 for BO+) and the predicted probabilities (ie, the continuous variable in step 3 calculated by the sigmoid function in the output layer in
Study flowchart. CNN: convolutional neural network; MPRSA: matching personal response scheme to adapt for the correct classification in the model.
The demographic data of the nurses are shown in
The highest in nurse hierarchy is N (132/1002, 13.2%), followed by N1 (134/1002, 13.4%), N2 (272/1002, 27.1%), N3 (248/1002, 24.8%), and N4 (215/1002, 21.5%). The top two job titles are nurse (797/1002, 79.5%) and leader (149/1002, 14.9%).
The average age for the sample is 32.6 (SD 7.2) years, ranging from 23 to 56. The average work experience in other hospitals reaches 15.1 (SD 28.5) months.
The workload in terms of the number of patients cared for in a week by each nurse averages 11 (SD 19.1). The mean for non-care affairs in a week reaches 4 hours (SD 5.8). The mean of nursing care is 9 (SD 2.7) hours per week. The average number of a patient cared for is 9 (SD 12.1).
Demographic data of the study sample.
Variable and type | Value | |
|
|
|
|
Male | 69 (6.9) |
|
Female | 933 (93.1) |
|
|
|
|
Less than university | 46 (4.6) |
|
University | 892 (89.0) |
|
Graduate school | 64 (6.4) |
|
|
|
|
Single | 596 (59.5) |
|
Married | 399 (39.8) |
|
Divorced | 7 (0.7) |
|
|
|
|
Without | 627 (62.6) |
|
With | 375 (37.4) |
|
|
|
|
N (<1 year experience) | 133 (13.3) |
|
N1 (Fundamentals of Nursing) | 134 (13.4) |
|
N2 (Critical Care in Nursing) | 272 (27.1) |
|
N3 (Holistic Care and Teaching) | 248 (24.8) |
|
N4 (Specialist Nursing and Research) | 215 (21.5) |
|
|
|
|
Nurse | 798 (79.6) |
|
Leader | 147 (14.7) |
|
Assistant head nurse | 30 (3.0) |
|
Head nurse | 27 (2.7) |
Age, mean (SD), range | 32.6 (7.2), 23-56 | |
Work experience outside hospital (month), mean (SD), range | 15.1 (28.5), 0-180 | |
Average hours spent in non-care affairs per week, mean (SD), range | 3.9 (5.8), 0-60 | |
Average weekly hours spent in nursing care, mean (SD), range | 9.2 (2.9), 1.5-70 | |
Average daily patient care, mean (SD), range | 9.5 (12.1), 0-120 |
A visual representation displaying the classification effect is plotted using the box plot (
Two study groups divided by the k-mean algorithm (A) and receiver operating characteristic curve (B).
The 20-item model yields a higher accuracy rate (0.95) with an AUC 0.98 (95% CI 0.97-1.00) higher than that of the 20-item model with an accuracy of 0.95 and an AUC 0.97 (95% CI 0.96-0.99) based on the 1002 cases.
The MPRSA applied to the bottom pattern in
The 700-case training set with 0.96 accuracies predicts the 302-case testing set reaching an accuracy of 0.91 (
Three scenarios applied to convolutional neural network for the prediction of nurse burnout (n=1002).
Sample | True condition | ||||
|
BO+a | BO–b | BO+/row # | BO–/row # | |
|
|
|
|
|
|
|
Positive | 507 | 26 | 0.95 | 0.05 |
|
Negative | 24 | 445 | 0.05 | 0.95 |
|
|
|
|
|
|
|
Positive | 531 | 0 | 1.00 | 0 |
|
Negative | 0 | 471 | 0 | 1.00 |
aBO+: suspicious for burnout.
bBO–: not suspicious for burnout.
cMPRSA: matching personal response scheme to adapt for the correct classification.
Training and testing effects.
Sample | True condition | ||||||||
|
BO+a | BO–b | BO+/row # | BO–/row # | |||||
|
|
|
|
|
|||||
|
Positive | 362 | 15 | 0.96 | 0.04 | ||||
|
Negative | 10 | 313 | 0.03 | 0.97 | ||||
|
|
|
|
|
|||||
|
Positive | 147 | 16 | 0.90 | 0.10 | ||||
|
Negative | 11 | 128 | 0.08 | 0.92 |
aBO+: suspicious for burnout.
bBO–: not suspicious for burnout.
An MBI-HSS app for nurses predicting BO was developed (
One resulting example is present in
Screenshot of the mobile phone app.
The result of assessing nurse burnout.
We observed that (1) the 20-item model yields a higher accuracy rate (0.95; AUC 0.97, 95% CI 0.94-0.95), (2) the MPRSA drives the model’s prior accuracy at 100%, (3) the 700-case training set with 0.96 accuracy predicts the 302-case testing set reaching an accuracy of 0.91, and (4) the MBI-HSS app for nurses predicting BO has been developed and demonstrated.
The MBI-HSS is the most widely used tool for measuring BO in the world [
Maslach and Jackson [
However, no matter which cutting point scheme is applied, that of Maslach and Jackson [
CNN can improve prediction accuracy (up to 7.14%) [
Over 708 articles have been found using the keyword “convolutional neural network” (Title) searched in PubMed Central on September 23, 2019. None used Microsoft Excel to perform the CNN. The interpretations for the CNN concept and process or the parameter estimations are shown in
Using Microsoft Excel to perform CNN is the third feature of this study (
Because the principle for concerning more with the vital few and less with the trivial numerous is emphasized in the quality control process, we propose the MPRSA as the fourth feature. We incorporated the original responses into the model to let the label be correctly classified by the CNN, through which all cases with a false prediction in the training set would be adjusted as a true prediction, reaching a 100% accuracy rate if the cases reoccur in the testing set.
Furthermore, the curves of category probabilities based on the Rasch rating scale model [
It is easy to set up the nurse BO online assessment if the designer uploads relevant and appropriate audio and visual files to the corresponding questions of the database. We applied the CNN algorithm along with the model’s parameters to design the routine on an app that is used to detect BO risk for nurses in hospitals (
As with all forms of web-based technology, advances in health communication technology are rapidly emerging [
The CNN module on Microsoft Excel is unique and innovative (
Our study has some limitations. First, although the psychometric properties of the 20-item MBI-HSS have been validated for measuring nurse BO in Taiwan [
Second, we have not discussed any improvement in predictive accuracy. For instance, whether other featured variables (eg, the mean, SD, and Lz index [
Third, the study was based on previously published [
Fourth, the MBI-HSS is a three-dimensional construct. Usually, the item difficulties should be first calibrated by using the Rasch ConQuest software [
Finally, the study sample was taken from Taiwanese data in a nurse survey. The model parameters estimated for the MBI-HSS Chinese version are only suitable for the Chinese (particularly for Taiwanese) society in health care settings. Generalizing these BO assessment findings (eg, the cutting point at around 43; see
We illustrate features and contributions in this study: (1) CNN performed in Microsoft Excel, (2) MPRSA applied to increase the model’s prior prediction accuracy, (3) an online app demonstrated to display results using a visual dashboard on Google Maps, and (4) the category probability curves based on Rasch rating scale model first observed in the CNN prediction model. The novelty of the app with the CNN algorithm improves the predictive accuracy of nurse BO. It is expected to help nurses self-assess job BO at an early stage in the future.
Study dataset.
Convolutional neural network to interpret Figure 1.
Mp4 for convolutional neural network performed in Excel.
area under the curve
burnout
convolutional neural network
Maslach Burnout Inventory–Human Services Survey
matching personal response scheme to adapt for the correct classification
personal accomplishment
receiver operation characteristic
YLL conceived and designed the study, WC and PHC performed the statistical analyses, and YTY was in charge of recruiting study participants. TWC helped design the study, collected information, and interpreted data. HFL monitored the research. All authors read and approved the final article.
None declared.