RefCheck Maintenance Notice

On Monday, December 3, 2018, from 16:00-18:00 EST, RefCheck will be undergoing maintenance. RefCheck is the process where, during copyediting, all references are extracted from the manuscript file, parsed, matched against various databases (eg, PubMed and CrossRef), and automatically corrected. For more information on RefCheck, please visit our Knowledge Base.

Who will be affected?


Journal Description

JMIR Medical Informatics (JMI, ISSN 2291-9694) (Editor-in-chief: Christian Lovis MD MPH FACMI) is a Pubmed/SCIE-indexed, top-rated, tier A journal with impact factor expected in 2019, which focuses on clinical informatics, big data in health and health care, decision support for health professionals, electronic health records, ehealth infrastructures and implementation. It has a focus on applied, translational research, with a broad readership including clinicians, CIOs, engineers, industry and health informatics professionals.

Published by JMIR Publications, publisher of the Journal of Medical Internet Research (JMIR), the leading eHealth/mHealth journal (Impact Factor 2018: 4.671), JMIR Med Inform has a slightly different scope (emphasizing more on applications for clinicians and health professionals rather than consumers/citizens, which is the focus of JMIR), publishes even faster, and also allows papers which are more technical or more formative than what would be published in the Journal of Medical Internet Research.

JMIR Medical Informatics is indexed in PubMed Central/PubMed, and has also been accepted for SCIE, with an official Clarivate impact factor 2018 expected to be released in 2019 (see announcement).

JMIR Medical Informatics adheres to the same quality standards as JMIR and all articles published here are also cross-listed in the Table of Contents of JMIR, the worlds' leading medical journal in health sciences / health services research and health informatics (


Recent Articles:

  • Source: Image created by the Authors; Copyright: Shahryar Eivazzadeh; URL:; License: Creative Commons Attribution + Noncommercial (CC-BY-NC).

    Most Influential Qualities in Creating Satisfaction Among the Users of Health Information Systems: Study in Seven European Union Countries


    Background: Several models suggest how the qualities of a product or service influence user satisfaction. Models such as the Customer Satisfaction Index (CSI), Technology Acceptance Model (TAM), and Delone and McLean Information Systems Success demonstrate those relations and have been used in the context of health information systems. Objective: This study aimed to investigate which qualities foster greater satisfaction among patient and professional users. In addition, we are interested in knowing to what extent improvement in those qualities can explain user satisfaction and whether this makes user satisfaction a proxy indicator of those qualities. Methods: The Unified eValuation using ONtology (UVON) method was used to construct an ontology of the required qualities for 7 electronic health (eHealth) apps being developed in the Future Internet Social and Technological Alignment Research (FI-STAR) project, a European Union (EU) project in electronic health (eHealth). The eHealth apps were deployed across 7 EU countries. The ontology included and unified the required qualities of those systems together with the aspects suggested by the Model for ASsessment of Telemedicine apps (MAST) evaluation framework. Moreover, 2 similar questionnaires for 87 patient users and 31 health professional users were elicited from the ontology. In the questionnaires, the user was asked if the system has improved the specified qualities and if the user was satisfied with the system. The results were analyzed using Kendall correlation coefficients matrices, incorporating the quality and satisfaction aspects. For the next step, 2 partial least squares structural equation modeling (PLS-SEM) path models were developed using the quality and satisfaction measure variables and the latent construct variables that were suggested by the UVON method. Results: Most of the quality aspects grouped by the UVON method are highly correlated. Strong correlations in each group suggest that the grouped qualities can be measures that reflect a latent quality construct. The PLS-SEM path analysis for the patients reveals that the effectiveness, safety, and efficiency of treatment provided by the system are the most influential qualities in achieving and predicting user satisfaction. For the professional users, effectiveness and affordability are the most influential. The parameters of the PLS-SEM that are calculated allow for the measurement of a user satisfaction index similar to CSI for similar health information systems. Conclusions: For both patients and professionals, the effectiveness of systems highly contributes to their satisfaction. Patients care about improvements in safety and efficiency, whereas professionals care about improvements in the affordability of treatments with health information systems. User satisfaction is reflected more in the users’ evaluation of system output and fulfillment of expectations but slightly less in how far the system is from ideal. Investigating satisfaction scores can be a simple and fast way to infer if the system has improved the abovementioned qualities in treatment and care.

  • Statistitical validation of clustering. Source: The Authors; Copyright: The Authors; URL:; License: Licensed by JMIR.

    Novel Approach to Cluster Patient-Generated Data Into Actionable Topics: Case Study of a Web-Based Breast Cancer Forum


    Background: The increasing use of social media and mHealth apps has generated new opportunities for health care consumers to share information about their health and well-being. Information shared through social media contains not only medical information but also valuable information about how the survivors manage disease and recovery in the context of daily life. Objective: The objective of this study was to determine the feasibility of acquiring and modeling the topics of a major online breast cancer support forum. Breast cancer patient support forums were selected to discover the hidden, less obvious aspects of disease management and recovery. Methods: First, manual topic categorization was performed using qualitative content analysis (QCA) of each individual forum board. Second, we requested permission from the Community for a more in-depth analysis of the postings. Topic modeling was then performed using open source software Machine Learning Language Toolkit, followed by multiple linear regression (MLR) analysis to detect highly correlated topics among the different website forums. Results: QCA of the forums resulted in 20 categories of user discussion. The final topic model organized >4 million postings into 30 manageable topics. Using qualitative analysis of the topic models and statistical analysis, we grouped these 30 topics into 4 distinct clusters with similarity scores of ≥0.80; these clusters were labeled Symptoms & Diagnosis, Treatment, Financial, and Family & Friends. A clinician review confirmed the clinical significance of the topic clusters, allowing for future detection of actionable items within social media postings. To identify the most significant topics across individual forums, MLR demonstrated that 6 topics—based on the Akaike information criterion values ranging from −642.75 to −412.32—were statistically significant. Conclusions: The developed method provides an insight into the areas of interest and concern, including those not ascertainable in the clinic. Such topics included support from lay and professional caregivers and late side effects of therapy that consumers discuss in social media and may be of interest to clinicians. The developed methods and results indicate the potential of social media to inform the clinical workflow with regards to the impact of recovery on daily life.

  • A multimedia display showing a peripherally inserted central venous catheter procedure. Source: The Authors; Copyright: The Authors; URL:; License: Licensed by JMIR.

    Effect of a Multimedia Patient Decision Aid to Supplement the Informed Consent Process of a Peripherally Inserted Central Venous Catheter Procedure: Pre-Post...


    Background: Informed consent is a complex process to help patients engage in care processes and reach the best treatment decisions. There are many limitations to the conventional consent process that is based on oral discussion of information related to treatment procedures by the health care provider. A conclusive body of research supports the effectiveness of multimedia patient decision aids (PtDAs) in the consent process in terms of patient satisfaction, increased knowledge about the procedure, reduced anxiety level, and higher engagement in the decision making. Little information is available about the effectiveness of multimedia PtDAs in the consent process of invasive therapeutic procedures such as the peripherally inserted central venous catheter (PICC). Objective: The objective of this study was to examine the effectiveness of a multimedia PtDA in supplementing the consent process of the PICC for patients in 10 acute and intensive care units in terms of knowledge recall, knowledge retention, satisfaction with the consent process, and satisfaction with the PICC multimedia PtDA. Methods: This pre-post quasi-experimental study included 130 patients for whom a PICC was ordered. Patients in the control group (n=65) received the conventional consent process for the PICC, while those in the intervention group (n=65) received the multimedia PtDA to support the consent process of a PICC. All patients were surveyed for knowledge recall and retention about the procedure and satisfaction with the consent process. Patients in the intervention group were also surveyed for their satisfaction with the multimedia PtDA. Results: Compared with the control group, patients in the intervention group scored around 2 points higher on knowledge recall (t125=4.9, P<.001) and knowledge retention (t126=4.8, P<.001). All patients in the intervention group were highly satisfied with the multimedia PtDA, with a mean score of >4.5 out of 5 on all items. Items with the highest mean scores were related to the effect of the multimedia PtDA on knowledge retention (mean 4.9 [SD 0.2]), patient readiness to learn (mean 4.8 [SD 0.5]), complete understanding of the procedure-related complications (mean 4.8 [SD 0.4]), and patient role in maintaining the safety of the PICC (mean 4.8 [SD 0.5]). Patients in the two groups were highly satisfied with the consent process. However, 15% (10/65) patients in the control group reported that the following information was omitted from the discussion: patient and provider roles in the safety of the PICC, other treatment options, and common side effects. Furthermore, 2 patients commented that they were not ready to engage in the discussion. Conclusions: The multimedia PtDA is an effective standardized, structured, self-paced learning tool to supplement the consent process of the PICC and improve patient satisfaction with the process, knowledge recall, and knowledge retention.

  • Lifeline user with breathing problems pressing her personal emergency button. Source: Philips Lifeline; Copyright: Philips Lifeline; URL:; License: Creative Commons Attribution (CC-BY).

    Predictive Modeling of 30-Day Emergency Hospital Transport of Patients Using a Personal Emergency Response System: Prognostic Retrospective Study


    Background: Telehealth programs have been successful in reducing 30-day readmissions and emergency department visits. However, such programs often focus on the costliest patients with multiple morbidities and last for only 30 to 60 days postdischarge. Inexpensive monitoring of elderly patients via a personal emergency response system (PERS) to identify those at high risk for emergency hospital transport could be used to target interventions and prevent avoidable use of costly readmissions and emergency department visits after 30 to 60 days of telehealth use. Objective: The objectives of this study were to (1) develop and validate a predictive model of 30-day emergency hospital transport based on PERS data; and (2) compare the model’s predictions with clinical outcomes derived from the electronic health record (EHR). Methods: We used deidentified medical alert pattern data from 290,434 subscribers to a PERS service to build a gradient tree boosting-based predictive model of 30-day hospital transport, which included predictors derived from subscriber demographics, self-reported medical conditions, caregiver network information, and up to 2 years of retrospective PERS medical alert data. We evaluated the model’s performance on an independent validation cohort (n=289,426). We linked EHR and PERS records for 1815 patients from a home health care program to compare PERS–based risk scores with rates of emergency encounters as recorded in the EHR. Results: In the validation cohort, 2.22% (6411/289,426) of patients had 1 or more emergency transports in 30 days. The performance of the predictive model of emergency hospital transport, as evaluated by the area under the receiver operating characteristic curve, was 0.779 (95% CI 0.774-0.785). Among the top 1% of predicted high-risk patients, 25.5% had 1 or more emergency hospital transports in the next 30 days. Comparison with clinical outcomes from the EHR showed 3.9 times more emergency encounters among predicted high-risk patients than low-risk patients in the year following the prediction date. Conclusions: Patient data collected remotely via PERS can be used to reliably predict 30-day emergency hospital transport. Clinical observations from the EHR showed that predicted high-risk patients had nearly four times higher rates of emergency encounters than did low-risk patients. Health care providers could benefit from our validated predictive model by targeting timely preventive interventions to high-risk patients. This could lead to overall improved patient experience, higher quality of care, and more efficient resource utilization.

  • Source: Flickr; Copyright: Alex Dodd; URL:; License: Creative Commons Attribution + Noncommercial + NoDerivatives (CC-BY-NC-ND).

    Extraction of Information Related to Adverse Drug Events from Electronic Health Record Notes: Design of an End-to-End Model Based on Deep Learning


    Background: Pharmacovigilance and drug-safety surveillance are crucial for monitoring adverse drug events (ADEs), but the main ADE-reporting systems such as Food and Drug Administration Adverse Event Reporting System face challenges such as underreporting. Therefore, as complementary surveillance, data on ADEs are extracted from electronic health record (EHR) notes via natural language processing (NLP). As NLP develops, many up-to-date machine-learning techniques are introduced in this field, such as deep learning and multi-task learning (MTL). However, only a few studies have focused on employing such techniques to extract ADEs. Objective: We aimed to design a deep learning model for extracting ADEs and related information such as medications and indications. Since extraction of ADE-related information includes two steps—named entity recognition and relation extraction—our second objective was to improve the deep learning model using multi-task learning between the two steps. Methods: We employed the dataset from the Medication, Indication and Adverse Drug Events (MADE) 1.0 challenge to train and test our models. This dataset consists of 1089 EHR notes of cancer patients and includes 9 entity types such as Medication, Indication, and ADE and 7 types of relations between these entities. To extract information from the dataset, we proposed a deep-learning model that uses a bidirectional long short-term memory (BiLSTM) conditional random field network to recognize entities and a BiLSTM-Attention network to extract relations. To further improve the deep-learning model, we employed three typical MTL methods, namely, hard parameter sharing, parameter regularization, and task relation learning, to build three MTL models, called HardMTL, RegMTL, and LearnMTL, respectively. Results: Since extraction of ADE-related information is a two-step task, the result of the second step (ie, relation extraction) was used to compare all models. We used microaveraged precision, recall, and F1 as evaluation metrics. Our deep learning model achieved state-of-the-art results (F1=65.9%), which is significantly higher than that (F1=61.7%) of the best system in the MADE1.0 challenge. HardMTL further improved the F1 by 0.8%, boosting the F1 to 66.7%, whereas RegMTL and LearnMTL failed to boost the performance. Conclusions: Deep learning models can significantly improve the performance of ADE-related information extraction. MTL may be effective for named entity recognition and relation extraction, but it depends on the methods, data, and other factors. Our results can facilitate research on ADE detection, NLP, and machine learning.

  • The P-R anonymization matrix (montage). Source: The Authors / Smartmockups; Copyright: JMIR Publications; URL:; License: Creative Commons Attribution (CC-BY).

    Contextual Anonymization for Secondary Use of Big Data in Biomedical Research: Proposal for an Anonymization Matrix


    Background: The current law on anonymization sets the same standard across all situations, which poses a problem for biomedical research. Objective: We propose a matrix for setting different standards, which is responsive to context and public expectations. Methods: The law and ethics applicable to anonymization were reviewed in a scoping study. Social science on public attitudes and research on technical methods of anonymization were applied to formulate a matrix. Results: The matrix adjusts anonymization standards according to the sensitivity of the data and the safety of the place, people, and projects involved. Conclusions: The matrix offers a tool with context-specific standards for anonymization in data research.

  • Leuven, Belgium. Source: Flickr; Copyright: Krijn van Putten; URL:; License: Creative Commons Attribution (CC-BY).

    Health Data for Research Through a Nationwide Privacy-Proof System in Belgium: Design and Implementation


    Background: Health data collected during routine care have important potential for reuse for other purposes, especially as part of a learning health system to advance the quality of care. Many sources of bias have been identified through the lifecycle of health data that could compromise the scientific integrity of these data. New data protection legislation requires research facilities to improve safety measures and, thus, ensure privacy. Objective: This study aims to address the question on how health data can be transferred from various sources and using multiple systems to a centralized platform, called, while ensuring the accuracy, validity, safety, and privacy. In addition, the study demonstrates how these processes can be used in various research designs relevant for learning health systems. Methods: The platform urges uniformity of the data registration at the primary source through the use of detailed clinical models. Data retrieval and transfer are organized through end-to-end encrypted electronic health channels, and data are encoded using token keys. In addition, patient identifiers are pseudonymized so that health data from the same patient collected across various sources can still be linked without compromising the deidentification. Results: The platform currently collects data for >150 clinical registries in Belgium. We demonstrated how the data collection for the Belgian primary care morbidity register INTEGO is organized and how the platform can be used for a cluster randomized trial. Conclusions: Collecting health data in various sources and linking these data to a single patient is a promising feature that can potentially address important concerns on the validity and quality of health data. Safe methods of data transfer without compromising privacy are capable of transporting these data from the primary data provider or clinician to a research facility. More research is required to demonstrate that these methods improve the quality of data collection, allowing researchers to rely on electronic health records as a valid source for scientific data.

  • An integrated decision support software for gay and bisexual men attending general practice (montage). Source: The Authors / Placeit; Copyright: JMIR Publications; URL:; License: Creative Commons Attribution (CC-BY).

    Assessing the Impacts of Integrated Decision Support Software on Sexual Orientation Recording, Comprehensive Sexual Health Testing, and Detection of...


    Background: Gay and bisexual men are disproportionately affected by HIV and other sexually transmissible infections (STIs), yet opportunities for sexual health testing of this population are often missed or incomplete in general practice settings. Strategies are needed for improving the uptake and completeness of sexual health testing in this setting. Objectives: The goal of the research was to evaluate the impact of an intervention centered around integrated decision support software and routine data feedback on the collection of sexual orientation data and sexual health testing among gay and bisexual men attending general practice. Methods: A study using before/after and intervention/comparison methods was undertaken to assess the intervention’s impact in 7 purposively sampled Australian general practice clinics located near the urban centers of Sydney and Melbourne. The software was introduced at staggered points between April and August 2012; it used patient records to prompt clinicians to record sexual orientation and accessed pathology testing history to generate prompts when sexual health testing was overdue or incomplete. The software also had a function for querying patient management system databases in order to generate de-identified data extracts, which were used to report regularly to participating clinicians. We calculated summary rate ratios (SRRs) based on quarterly trends and used Poisson regression analyses to assess differences between the 12-month preintervention and 24-month intervention periods as well as between the intervention sites and 4 similar comparison sites that did not receive the intervention. Results: Among 32,276 male patients attending intervention clinics, sexual orientation recording increased 19% (from 3213/6909 [46.50%] to 5136/9110 [56.38%]) during the intervention period (SRR 1.10, 95% CI 1.04-1.11, P<.001) while comprehensive sexual health testing increased by 89% (305/1159 [26.32%] to 690/1413 [48.83%]; SRR 1.38, 95% CI 1.28-1.46, P<.001). Comprehensive testing increased slightly among the 7290 gay and bisexual men attending comparison sites, but the increase was comparatively greater in clinics that received the intervention (SRR 1.12, 95% CI 1.10-1.14, P<.001). In clinics that received the intervention, there was also an increase in detection of chlamydia and gonorrhea that was not observed in the comparison sites. Conclusions: Integrated decision support software and data feedback were associated with modest increases in sexual orientation recording, comprehensive testing among gay and bisexual men, and the detection of STIs. Tests for and detection of chlamydia and gonorrhea were the most dramatically impacted. Decision support software can be used to enhance the delivery of sexual health care in general practice.

  • Source: Pixabay; Copyright: Sasin Tipchai; URL:; License: Public Domain (CC0).

    Identifying Patients Who Are Likely to Receive Most of Their Care From a Specific Health Care System: Demonstration via Secondary Analysis


    Background: In the United States, health care is fragmented in numerous distinct health care systems including private, public, and federal organizations like private physician groups and academic medical centers. Many patients have their complete medical data scattered across these several health care systems, with no particular system having complete data on any of them. Several major data analysis tasks such as predictive modeling using historical data are considered impractical on incomplete data. Objective: Our objective was to find a way to enable these analysis tasks for a health care system with incomplete data on many of its patients. Methods: This study presents, to the best of our knowledge, the first method to use a geographic constraint to identify a reasonably large subset of patients who tend to receive most of their care from a given health care system. A data analysis task needing relatively complete data can be conducted on this subset of patients. We demonstrated our method using data from the University of Washington Medicine (UWM) and PreManage data covering the use of all hospitals in Washington State. We compared 10 candidate constraints to optimize the solution. Results: For UWM, the best constraint is that the patient has a UWM primary care physician and lives within 5 miles of at least one UWM hospital. About 16.01% (55,707/348,054) of UWM patients satisfied this constraint. Around 69.38% (10,501/15,135) of their inpatient stays and emergency department visits occurred within UWM in the following 6 months, more than double the corresponding percentage for all UWM patients. Conclusions: Our method can identify a reasonably large subset of patients who tend to receive most of their care from UWM. This enables several major analysis tasks on incomplete medical data that were previously deemed infeasible.

  • Emergency department. Source: FEMA; Copyright: Robert Kaufmann / FEMA; URL:; License: Public Domain (CC0).

    Appropriateness of Hospital Admission for Emergency Department Patients with Bronchiolitis: Secondary Analysis


    Background: Bronchiolitis is the leading cause of hospitalization in children under 2 years of age. Each year in the United States, bronchiolitis results in 287,000 emergency department visits, 32%-40% of which end in hospitalization. Frequently, emergency department disposition decisions (to discharge or hospitalize) are made subjectively because of the lack of evidence and objective criteria for bronchiolitis management, leading to significant practice variation, wasted health care use, and suboptimal outcomes. At present, no operational definition of appropriate hospital admission for emergency department patients with bronchiolitis exists. Yet, such a definition is essential for assessing care quality and building a predictive model to guide and standardize disposition decisions. Our prior work provided a framework of such a definition using 2 concepts, one on safe versus unsafe discharge and another on necessary versus unnecessary hospitalization. Objective: The goal of this study was to determine the 2 threshold values used in the 2 concepts, with 1 value per concept. Methods: Using Intermountain Healthcare data from 2005-2014, we examined distributions of several relevant attributes of emergency department visits by children under 2 years of age for bronchiolitis. Via a data-driven approach, we determined the 2 threshold values. Results: We completed the first operational definition of appropriate hospital admission for emergency department patients with bronchiolitis. Appropriate hospital admissions include actual admissions with exposure to major medical interventions for more than 6 hours, as well as actual emergency department discharges, followed by an emergency department return within 12 hours ending in admission for bronchiolitis. Based on the definition, 0.96% (221/23,125) of the emergency department discharges were deemed unsafe. Moreover, 14.36% (432/3008) of the hospital admissions from the emergency department were deemed unnecessary. Conclusions: Our operational definition can define the prediction target for building a predictive model to guide and improve emergency department disposition decisions for bronchiolitis in the future.

  • Paper vital sign records. Source: Pixabay; Copyright: rawpixel; URL:; License: Public Domain (CC0).

    Impact of Electronic Versus Paper Vital Sign Observations on Length of Stay in Trauma Patients: Stepped-Wedge, Cluster Randomized Controlled Trial


    Background: Electronic recording of vital sign observations (e-Obs) has become increasingly prevalent in hospital care. The evidence of clinical impact for these systems is mixed. Objective: The objective of our study was to assess the effect of e-Obs versus paper documentation (paper) on length of stay (time between trauma unit admission and “fit to discharge”) for trauma patients. Methods: A single-center, randomized stepped-wedge study of e-Obs against paper was conducted in two 26-bed trauma wards at a medium-sized UK teaching hospital. Randomization of the phased intervention order to 12 study areas was computer generated. The primary outcome was length of stay. Results: A total of 1232 patient episodes were randomized (paper: 628, e-Obs: 604). There were 37 deaths in hospital: 21 in the paper arm and 16 in the e-Obs arm. For discharged patients, the median length of stay was 5.4 (range: 0.2-79.0) days on the paper arm and 5.6 (range: 0.1-236.7) days on the e-Obs arm. Competing risks regression analysis for time to discharge showed no difference between the treatment arms (subhazard ratio: 1.05; 95% CI 0.82-1.35; P=.68). A greater proportion of patient episodes contained an Early Warning Score (EWS) ≥3 using the e-Obs system than using paper (subhazard ratio: 1.63; 95% CI 1.28-2.09; P<.001). However, there was no difference in the time to the subsequent observation, “escalation time” (hazard ratio 1.05; 95% CI 0.80-1.38; P=.70). Conclusions: The phased introduction of an e-Obs documentation system was not associated with a change in length of stay. A greater proportion of patient episodes contained an EWS≥3 using the e-Obs system, but this was not associated with a change in “escalation time.” Trial Registration: ISRCTN Registry ISRCTN91040762; (Archived by WebCite at

  • Source: Pixabay; Copyright: Riala; URL:; License: Public Domain (CC0).

    Benefits and Costs of Digital Consulting in Clinics Serving Young People With Long-Term Conditions: Mixed-Methods Approach


    Background: Since the introduction of digital health technologies in National Health Service (NHS), health professionals are starting to use email, text, and other digital methods to consult with their patients in a timely manner. There is lack of evidence regarding the economic impact of digital consulting in the United Kingdom (UK) NHS. Objective: This study aimed to estimate the direct costs associated with digital consulting as an adjunct to routine care at 18 clinics serving young people aged 16-24 years with long-term conditions. Methods: This study uses both quantitative and qualitative approaches. Semistructured interviews were conducted with 173 clinical team members on the impacts of digital consulting. A structured questionnaire was developed and used for 115 health professionals across 12 health conditions at 18 sites in the United Kingdom to collect data on time and other resources used for digital consulting. A follow-up semistructured interview was conducted with a single senior clinician at each site to clarify the mechanisms through which digital consulting use might lead to outcomes relevant to economic evaluation. We used the two-part model to see the association between the time spent on digital consulting and the job role of staff, type of clinic, and the average length of the working hours using digital consulting. Results: When estimated using the two-part model, consultants spent less time on digital consulting compared with nurses (95.48 minutes; P<.001), physiotherapists (55.3 minutes; P<.001), and psychologists (31.67 minutes; P<.001). Part-time staff spent less time using digital consulting than full-time staff despite insignificant result (P=.15). Time spent on digital consulting differed across sites, and no clear pattern in using digital consulting was found. Health professionals qualitatively identified the following 4 potential economic impacts for the NHS: decreasing adverse events, improving patient well-being, decreasing wait lists, and staff workload. We did not find evidence to suggest that the clinical condition was associated with digital consulting use. Conclusions: Nurses and physiotherapists were the greatest users of digital consulting. Teams appear to use an efficient triage system with the most expensive members digitally consulting less than lower-paid team members. Staff report showed concerns regarding time spent digitally consulting, which implies that direct costs increase. There remain considerable gaps in evidence related to cost-effectiveness of digital consulting, but this study has highlighted important cost-related outcomes for assessment in future cost-effectiveness trials of digital consulting.

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Latest Submissions Open for Peer-Review:

View All Open Peer Review Articles
  • E-consultation in primary care: A systematic review

    Date Submitted: Dec 6, 2018

    Open Peer Review Period: Dec 11, 2018 - Feb 5, 2019

    Background: Background: Governments and healthcare providers are keen to find innovative ways to more efficiently deliver care. Interest in e-consultation has grown, but evidence of benefit is uncerta...

    Background: Background: Governments and healthcare providers are keen to find innovative ways to more efficiently deliver care. Interest in e-consultation has grown, but evidence of benefit is uncertain. Objective: Aims: To assess the evidence of delivering e-consultation using secure email/messaging or video links in primary care. Methods: Design & Methods: A systematic review was conducted focusing on the use and application of e-consultations in primary care. A systematic review of seven international databases was searched (Medline, Embase, CINAHL, Cochrane Library, PsycINFO, Econlit and Web of Science) (1999-2017), identifying 52 relevant studies. Screening was conducted against a detailed inclusion and exclusion criteria. Independent dual data extraction was conducted and assessed for quality. The resulting evidence was synthesised using thematic analysis. Results: Results: Patient responses to e-consultation are mixed. Patients report satisfaction with services, and improved self-care, communication and engagement with clinicians. Evidence for the acceptability and ease of use was strong, especially for those with long-term conditions and patients located in remote regions. However, patients were concerned about the privacy and security of their records. For primary healthcare staff, e-consultation delivers challenges around time-management, having the correct technological infrastructure, whether it offers a comparable standard of clinical quality, and whether it impacts on health outcomes. Conclusions: Conclusion: The emerging evidence around e-consultation is promising, with patients especially those with long-term conditions, finding this method of delivery useful. The small, pilot nature of many of the studies and low adoption rates result in unanswered questions about usage, quality, cost and sustainability. Clinical Trial: PROSPERO (International Prospective Register of Systematic Reviews) Registration Number: CRD42015019152

  • Statistical Natural Language Processing Methods for the Extraction of Geriatric Syndromes from Electronic Health Record Clinical Notes

    Date Submitted: Dec 10, 2018

    Open Peer Review Period: Dec 10, 2018 - Dec 20, 2018

    Background: Geriatric syndromes in older adults are associated with adverse outcomes. However, despite being reported in clinical notes these syndromes are often poorly captured by diagnostic codes in...

    Background: Geriatric syndromes in older adults are associated with adverse outcomes. However, despite being reported in clinical notes these syndromes are often poorly captured by diagnostic codes in the structured fields of electronic health records (EHRs) or administrative records. Objective: We aim to automatically determine if a patient has any geriatric syndromes by mining the free text of associated EHR clinical notes. We assessed which statistical natural language processing (NLP) techniques are most effective. Methods: We applied Conditional Random Fields (CRFs), a widely used machine learning algorithm, to identify each of 10 geriatric syndrome constructs in a clinical note. We assessed three sets of features/attributes for CRF operations: a base set, enhanced token, and contextual features. We trained the CRF on 3901 manually annotated notes from 85 patients, tuned the CRF on a validation set of 50 patients, and evaluated it on 50 held-out test patients. These notes were from a group of US Medicare (over 65) patients enrolled in a "Medicare-Advantage" HMO and cared for by a large group practice in Massachusetts. Results: A final feature set was formed through comprehensive feature ablation experiments. The final CRF model performed well at patient-level determination (macro-F1=0.834, micro-F1=0.851); however, performance varied by construct. For example, at phrase-partial evaluation, CRF worked well on constructs like absence of fecal control (F1=0.857) and vision impairment (F1=0.798), but poorly on malnutrition (F1=0.155), weight loss (F1=0.394) and severe urinary control issues (F1=0.532). Errors were primarily due to previously unobserved words (out-of-vocabulary) and a lack of context. Conclusions: This study shows that statistical NLP can be used to identify geriatric syndromes from EHR-extracted clinical notes. This creates new opportunities to identify patients with geriatric syndromes and study their health outcomes.

  • Survey of Electronic Health Record (EHR) Systems in Kenyan Public Hospitals: A mixed-methods survey

    Date Submitted: Dec 3, 2018

    Open Peer Review Period: Dec 7, 2018 - Feb 1, 2019

    Background: As healthcare facilities in Low- and Middle-Income Countries (LMICs) such as Kenya adopt Electronic Health Record (EHR) systems to improve hospital administration and patient care, it is i...

    Background: As healthcare facilities in Low- and Middle-Income Countries (LMICs) such as Kenya adopt Electronic Health Record (EHR) systems to improve hospital administration and patient care, it is important to understand the adoption process, identify the key stakeholders, and assess the capabilities of the systems in use. Objective: To describe the level of adoption of Electronic Health Records systems in public hospitals and understand the process of adoption from Health Management Information System (HMIS) system vendors and system users. Methods: We conducted a survey of County Health Records Information Officers (CHRIOs) in Kenya to determine the level of adoption of Electronic Health Records systems in public hospitals. We conducted site visits to hospitals to view systems in use and to interview hospital administrators and end users. We also interviewed Health Management Information System (HMIS) system vendors to understand the adoption process from their perspective. Results: From the survey of CHRIOs, all facilities mentioned had adopted some form of EHR. Hospitals commonly purchased systems for patient administration and hospital billing functions. Radiology and laboratory management systems were commonly standalone systems. There were varying levels of interoperability within facilities that had more than one system in operation. We only saw one in-patient EHR system in use although many vendors and hospital administrators we interviewed were planning to adopt or support such systems. From the user perspective, issues such as system usability, adequate training, availability of adequate infrastructure and system support emerged. From the vendor perspective, a wide range of services was available to the hospital though constrained by funding and the need to computerise service areas that were deemed as priority. Additionally, vendors were unable to implement some data sharing modules linking to national HMIS due to lack of appropriate policies to facilitate this and users’ lack of confidence in new technologies such as cloud services. Conclusions: EHR adoption in Kenya has been underway for some years, particularly in comprehensive care clinics, and hospitals are increasing purchasing systems to support administrative functions. Considerable support from government, donors and regional health informatics organisations will be required to enable hospitals to move to full EHR adoption for in-patient care.

  • Reinforcement Learning Based Method for Managing Type 1 Diabetes

    Date Submitted: Nov 24, 2018

    Open Peer Review Period: Dec 3, 2018 - Jan 28, 2019

    Background: Diabetes is a serious chronic disease marked by high levels of blood glucose. It results from issues related to how insulin is produced and/or how insulin functions in the body. In the lon...

    Background: Diabetes is a serious chronic disease marked by high levels of blood glucose. It results from issues related to how insulin is produced and/or how insulin functions in the body. In the long run, uncontrolled blood sugar can damage the vessels that supply blood to important organs such as heart, kidneys, eyes, and nerves. Currently there are no effective algorithms to automatically recommend insulin dosage level considering the characteristics of a diabetic patient. Objective: The objective of this work is to develop and validate a general reinforcement learning framework and a related learning model for personalized treatment and management of Type 1 diabetes and its complications. Methods: This research presents a model-free reinforcement learning (RL) algorithm to recommend insulin level to regulate the blood glucose level of a diabetic patient considering his/her state defined by A1C level, alcohol usage, activity level, and BMI value. In this approach, an RL agent learns from its exploration and response of diabetic patients when they are subject to different actions in terms of insulin dosage level. As a result of a treatment action at time step t, the RL agent receives a numeric reward depending on the response of the patient’s blood glucose level. At each stage the reward for the learning agent is calculated as a function of the difference between the glucose level in the patient body and its target level. The RL algorithm is trained on ten years of the clinical data of 87 patients obtained from the Mass General Hospital. Demographically, 59% of patients are male and 41% of patients are female; the median of age is 54 years and mean is 52.92 years; 86% of patients are white and 47% of 87 patients are married. Results: The performance of the algorithm is evaluated on 60 test cases. Further the performance of Support Vector Machine (SVM) has been applied for Lantus class prediction and results has been compared with Q-learning algorithm recommendation. The results show that the RL recommendations of insulin levels for test patients match with the actual prescriptions of the test patients. The RL gave prediction with an accuracy of 88% and SVM shows 80% accuracy. Conclusions: Since the RL algorithm can select actions that improve patient condition by taking into account delayed effects, it has a good potential to control blood glucose level in diabetic patients.

  • Evidence-Based Physical Therapy Practice in the State of Kuwait: Attitudes, Beliefs, Knowledge, Skills, and Barriers.

    Date Submitted: Nov 12, 2018

    Open Peer Review Period: Dec 3, 2018 - Jan 28, 2019

    Background: Evidence-based practice (EBP) is necessary to improve the practice of physical therapy (PT). However, a lack of knowledge and skills among physical therapists (PTs) and the presence of bar...

    Background: Evidence-based practice (EBP) is necessary to improve the practice of physical therapy (PT). However, a lack of knowledge and skills among physical therapists (PTs) and the presence of barriers may hinder the implementation of EBP in the State of Kuwait. Objective: The objectives of this study were to extensively 1) investigate attitudes toward EBP, 2) assess the current level of knowledge and skills necessary for EBP, and 3) identify the barriers to EBP among PTs in the State of Kuwait. Methods: The following methods were used: 1) a previously validated self-reported questionnaire and 2) a face-to-face semi-structured interview. The questionnaire, which was distributed to 200 PTs, examined the attitudes and beliefs of PTS about EBP; the interest in and motivation to engage in EBP; educational background, knowledge and skills related to accessing and interpreting information; level of attention to and use of the literature; access to and availability of information to promote EBP; and the perceived barriers to using EBP. The interview explored the factors that promote or discourage EBP. Descriptive statistics and logistic regression analyses were used. Results: Of the 200 non-randomly distributed questionnaires, 184 (92%) were completed and returned. In general, the PTs had positive attitudes, beliefs, and interests in EBP. Their educational background knowledge and skills related to assessing and interpreting information were well founded. The top three barriers included insufficient time (59%), lack of information resources (49%) and inapplicability of the research findings to the patient population (40.6%). Conclusions: EBP lacks support from superiors at work. Thus, identifying methods and strategies to support PTs in adopting EBP in the State of Kuwait is necessary.

  • Evaluation of Nursing Information System Implementation in two General Hospitals Affiliated to Zahedan University of Medical Sciences in Southeast Iran: Nurses’ Viewpoints

    Date Submitted: Nov 4, 2018

    Open Peer Review Period: Nov 8, 2018 - Jan 3, 2019

    Background: The use of NIS capabilities depends on the needs of users. Also, the proper design of these systems disrupts the daily processes of the users and complicates the acceptance of these system...

    Background: The use of NIS capabilities depends on the needs of users. Also, the proper design of these systems disrupts the daily processes of the users and complicates the acceptance of these systems. Objective: the purpose of this study was to evaluate nurses’ perceptions about the effectiveness of the NIS system and its impact on their activities. Methods: This cross-sectional survey was conducted in 2017. The research population consisted of 656 nurses working in two general hospitals affiliated to Zahedan University of Medical Sciences. According to the Cochran formula, 346 people were needed and a random stratified sampling was used to select the sample in each hospital. Data collection tool, model and questionnaire designed by Hung-Hsiou Hsu et al. The questionnaire consisted of two parts: the first part contained demographic information, and the second part was designed to collect nursing staff's views on the implementation of the NIS system. Validity of the questionnaire was verified by five experts. To determine the reliability of the questionnaire, a re-validation test was used. The Cronbach's alpha of the questionnaire was 0.92. Data analysis was done using SPSS.v22 software. Results: The highest and lowest mean scores of nurses' perceptions related to the ease of use perceived and user satisfaction with the score of 3.55 ± 67 and 3.33 ± 1.39 respectively. According to the regression test, the quality of information and service quality have a positive effect on the perceived ease of use and perceived usefulness of the NIS. And the quality of the system does not affect the perceived usefulness of the NIS. The perceived usefulness of using NIS has a positive and significant impact on users' willingness to use this system and user satisfaction over their intention to use of NIS. Conclusions: The lack of complete satisfaction of NIS users in this study could be due to the lack of user-friendliness, ease of use, and the inability to receive timely information for nurses. One of the obvious weaknesses of this system is from the viewpoint of nurses that there are unnecessary items that somehow slow down the work process. It seems that the needs of users in the design or purchase of NIS are not completely covered and new requirements are not taken into account during the use of this system.