TY - JOUR AU - Riou, Christine AU - El Azzouzi, Mohamed AU - Hespel, Anne AU - Guillou, Emeric AU - Coatrieux, Gouenou AU - Cuggia, Marc PY - 2025/4/17 TI - Ensuring General Data Protection Regulation Compliance and Security in a Clinical Data Warehouse From a University Hospital: Implementation Study JO - JMIR Med Inform SP - e63754 VL - 13 KW - clinical data warehouse KW - privacy KW - personal data protection KW - legislation KW - security KW - compliance KW - personal data KW - applicability KW - experiential analysis KW - university hospitals KW - French KW - France KW - data hub KW - operational challenge N2 - Background: The European Union?s General Data Protection Regulation (GDPR) has profoundly influenced health data management, with significant implications for clinical data warehouses (CDWs). In 2021, France pioneered a national framework for GDPR-compliant CDW implementation, established by its data protection authority (Commission Nationale de l?Informatique et des Libertés). This framework provides detailed guidelines for health care institutions, offering a unique opportunity to assess practical GDPR implementation in health data management. Objective: This study evaluates the real-world applicability of France?s CDW framework through its implementation at a major university hospital. It identifies practical challenges for its implementation by health institutions and proposes adaptations relevant to regulatory authorities in order to facilitate research in secondary use data domains. Methods: A systematic assessment was conducted in May 2023 at the University Hospital of Rennes, which manages data for over 2 million patients through the eHOP CDW system. The evaluation examined 116 criteria across 13 categories using a dual-assessment approach validated by information security and data protection officers. Compliance was rated as met, unmet, or not applicable, with criteria classified as software-related (n=25) or institution-related (n=91). Results: Software-related criteria showed 60% (n=15) compliance, with 28% (n=7) noncompliant or partially compliant and 12% (n=3) not applicable. Institution-related criteria achieved 72% (n=28) compliance for security requirements. Key challenges included managing genetic data, implementing automated archiving, and controlling data exports. The findings revealed effective privacy protection measures but also highlighted areas requiring regulatory adjustments to better support research. Conclusions: This first empirical assessment of a national CDW compliance framework offers valuable insights for health care institutions implementing GDPR requirements. While the framework establishes robust privacy protections, certain provisions may overly constrain research activities. The study identifies opportunities for framework evolution, balancing data protection with research imperatives. UR - https://medinform.jmir.org/2025/1/e63754 UR - http://dx.doi.org/10.2196/63754 ID - info:doi/10.2196/63754 ER - TY - JOUR AU - Sumsion, Daniel AU - Davis, Elijah AU - Fernandes, Marta AU - Wei, Ruoqi AU - Milde, Rebecca AU - Veltink, Malou Jet AU - Kong, Wan-Yee AU - Xiong, Yiwen AU - Rao, Samvrit AU - Westover, Tara AU - Petersen, Lydia AU - Turley, Niels AU - Singh, Arjun AU - Buss, Stephanie AU - Mukerji, Shibani AU - Zafar, Sahar AU - Das, Sudeshna AU - Junior, Moura Valdery AU - Ghanta, Manohar AU - Gupta, Aditya AU - Kim, Jennifer AU - Stone, Katie AU - Mignot, Emmanuel AU - Hwang, Dennis AU - Trotti, Marie Lynn AU - Clifford, D. Gari AU - Katwa, Umakanth AU - Thomas, Robert AU - Westover, Brandon M. AU - Sun, Haoqi PY - 2025/4/10 TI - Identification of Patients With Congestive Heart Failure From the Electronic Health Records of Two Hospitals: Retrospective Study JO - JMIR Med Inform SP - e64113 VL - 13 KW - electronic health record KW - machine learning KW - artificial intelligence KW - phenotype KW - congestive heart failure KW - medication KW - claims database KW - International Classification of Diseases KW - effectiveness KW - natural language processing KW - model performance KW - logistic regression KW - validity N2 - Background: Congestive heart failure (CHF) is a common cause of hospital admissions. Medical records contain valuable information about CHF, but manual chart review is time-consuming. Claims databases (using International Classification of Diseases [ICD] codes) provide a scalable alternative but are less accurate. Automated analysis of medical records through natural language processing (NLP) enables more efficient adjudication but has not yet been validated across multiple sites. Objective: We seek to accurately classify the diagnosis of CHF based on structured and unstructured data from each patient, including medications, ICD codes, and information extracted through NLP of notes left by providers, by comparing the effectiveness of several machine learning models. Methods: We developed an NLP model to identify CHF from medical records using electronic health records (EHRs) from two hospitals (Mass General Hospital and Beth Israel Deaconess Medical Center; from 2010 to 2023), with 2800 clinical visit notes from 1821 patients. We trained and compared the performance of logistic regression, random forests, and RoBERTa models. We measured model performance using area under the receiver operating characteristic curve (AUROC) and area under the precision-recall curve (AUPRC). These models were also externally validated by training the data on one hospital sample and testing on the other, and an overall estimated error was calculated using a completely random sample from both hospitals. Results: The average age of the patients was 66.7 (SD 17.2) years; 978 (54.3%) out of 1821 patients were female. The logistic regression model achieved the best performance using a combination of ICD codes, medications, and notes, with an AUROC of 0.968 (95% CI 0.940-0.982) and an AUPRC of 0.921 (95% CI 0.835-0.969). The models that only used ICD codes or medications had lower performance. The estimated overall error rate in a random EHR sample was 1.6%. The model also showed high external validity from training on Mass General Hospital data and testing on Beth Israel Deaconess Medical Center data (AUROC 0.927, 95% CI 0.908-0.944) and vice versa (AUROC 0.968, 95% CI 0.957-0.976). Conclusions: The proposed EHR-based phenotyping model for CHF achieved excellent performance, external validity, and generalization across two institutions. The model enables multiple downstream uses, paving the way for large-scale studies of CHF treatment effectiveness, comorbidities, outcomes, and mechanisms. UR - https://medinform.jmir.org/2025/1/e64113 UR - http://dx.doi.org/10.2196/64113 UR - http://www.ncbi.nlm.nih.gov/pubmed/40208662 ID - info:doi/10.2196/64113 ER - TY - JOUR AU - Remaki, Adam AU - Ung, Jacques AU - Pages, Pierre AU - Wajsburt, Perceval AU - Liu, Elise AU - Faure, Guillaume AU - Petit-Jean, Thomas AU - Tannier, Xavier AU - Gérardin, Christel PY - 2025/4/9 TI - Improving Phenotyping of Patients With Immune-Mediated Inflammatory Diseases Through Automated Processing of Discharge Summaries: Multicenter Cohort Study JO - JMIR Med Inform SP - e68704 VL - 13 KW - secondary use of clinical data for research and surveillance KW - clinical informatics KW - clinical data warehouse KW - electronic health record KW - data science KW - artificial intelligence KW - AI KW - natural language processing KW - ontologies KW - classifications KW - coding KW - tools KW - programs and algorithms KW - immune-mediated inflammatory diseases N2 - Background: Valuable insights gathered by clinicians during their inquiries and documented in textual reports are often unavailable in the structured data recorded in electronic health records (EHRs). Objective: This study aimed to highlight that mining unstructured textual data with natural language processing techniques complements the available structured data and enables more comprehensive patient phenotyping. A proof-of-concept for patients diagnosed with specific autoimmune diseases is presented, in which the extraction of information on laboratory tests and drug treatments is performed. Methods: We collected EHRs available in the clinical data warehouse of the Greater Paris University Hospitals from 2012 to 2021 for patients hospitalized and diagnosed with 1 of 4 immune-mediated inflammatory diseases: systemic lupus erythematosus, systemic sclerosis, antiphospholipid syndrome, and Takayasu arteritis. Then, we built, trained, and validated natural language processing algorithms on 103 discharge summaries selected from the cohort and annotated by a clinician. Finally, all discharge summaries in the cohort were processed with the algorithms, and the extracted data on laboratory tests and drug treatments were compared with the structured data. Results: Named entity recognition followed by normalization yielded F1-scores of 71.1 (95% CI 63.6-77.8) for the laboratory tests and 89.3 (95% CI 85.9-91.6) for the drugs. Application of the algorithms to 18,604 EHRs increased the detection of antibody results and drug treatments. For instance, among patients in the systemic lupus erythematosus cohort with positive antinuclear antibodies, the rate increased from 18.34% (752/4102) to 71.87% (2949/4102), making the results more consistent with the literature. Conclusions: While challenges remain in standardizing laboratory tests, particularly with abbreviations, this work, based on secondary use of clinical data, demonstrates that automated processing of discharge summaries enriched the information available in structured data and facilitated more comprehensive patient profiling. UR - https://medinform.jmir.org/2025/1/e68704 UR - http://dx.doi.org/10.2196/68704 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/68704 ER - TY - JOUR AU - Cox, Samuel AU - Masood, Erum AU - Panagi, Vasiliki AU - Macdonald, Calum AU - Milligan, Gordon AU - Horban, Scott AU - Santos, Roberto AU - Hall, Chris AU - Lea, Daniel AU - Tarr, Simon AU - Mumtaz, Shahzad AU - Akashili, Emeka AU - Rae, Andy AU - Urwin, Esmond AU - Cole, Christian AU - Sheikh, Aziz AU - Jefferson, Emily AU - Quinlan, Roy Philip PY - 2025/4/2 TI - Conversion of Sensitive Data to the Observational Medical Outcomes Partnership Common Data Model: Protocol for the Development and Use of Carrot JO - JMIR Res Protoc SP - e60917 VL - 14 KW - data standardization KW - OMOP KW - Observational Medical Outcomes Partnership KW - ETL KW - extract, transform, and load KW - data discovery KW - transparency KW - Carrot tool KW - common data model KW - data standard KW - health care KW - data model KW - data protection KW - data privacy KW - open-source N2 - Background: The use of data standards is low across the health care system, and converting data to a common data model (CDM) is usually required to undertake international research. One such model is the Observational Medical Outcomes Partnership (OMOP) CDM. It has gained substantial traction across researchers and those who have developed data platforms. The Observational Health Care Data Sciences and Informatics (OHDSI) partnership manages OMOP and provides many open-source tools to assist in converting data to the OMOP CDM. The challenge, however, is in the skills, knowledge, know-how, and capacity within teams to convert their data to OMOP. The European Health Care Data Evidence Network provided funds to allow data owners to bring in external resources to do the required conversions. The Carrot software (University of Nottingham) is a new set of open-source tools designed to help address these challenges while not requiring data access by external resources. Objective: The use of data protection rules is increasing, and privacy by design is a core principle under the European and UK legislations related to data protection. Our aims for the Carrot software were to have a standardized mechanism for managing the data curation process, capturing the rules used to convert the data, and creating a platform that can reuse rules across projects to drive standardization of process and improve the speed without compromising on quality. Most importantly, we aimed to deliver this design-by-privacy approach without requiring data access to those creating the rules. Methods: The software was developed using Agile approaches by both software engineers and data engineers, who would ultimately use the system. Experts in OMOP were used to ensure the approaches were correct. An incremental release program was initiated to ensure we delivered continuous progress. Results: Carrot has been delivered and used on a project called COVID-Curated and Open Analysis and Research Platform (CO-CONNECT) to assist in the process of allowing datasets to be discovered via a federated platform. It has been used to create over 45,000 rules, and over 5 million patient records have been converted. This has been achieved while maintaining our principle of not allowing access to the underlying data by the team creating the rules. It has also facilitated the reuse of existing rules, with most rules being reused rather than manually curated. Conclusions: Carrot has demonstrated how it can be used alongside existing OHDSI tools with a focus on the mapping stage. The COVID-Curated and Open Analysis and Research Platform project successfully managed to reuse rules across datasets. The approach is valid and brings the benefits expected, with future work continuing to optimize the generation of rules. International Registered Report Identifier (IRRID): RR1-10.2196/60917 UR - https://www.researchprotocols.org/2025/1/e60917 UR - http://dx.doi.org/10.2196/60917 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/60917 ER - TY - JOUR AU - West, Matthew AU - Cheng, You AU - He, Yingnan AU - Leng, Yu AU - Magdamo, Colin AU - Hyman, T. Bradley AU - Dickson, R. John AU - Serrano-Pozo, Alberto AU - Blacker, Deborah AU - Das, Sudeshna PY - 2025/3/31 TI - Unsupervised Deep Learning of Electronic Health Records to Characterize Heterogeneity Across Alzheimer Disease and Related Dementias: Cross-Sectional Study JO - JMIR Aging SP - e65178 VL - 8 KW - Alzheimer disease and related dementias KW - electronic health records KW - large language models KW - clustering KW - unsupervised learning N2 - Background: Alzheimer disease and related dementias (ADRD) exhibit prominent heterogeneity. Identifying clinically meaningful ADRD subtypes is essential for tailoring treatments to specific patient phenotypes. Objective: We aimed to use unsupervised learning techniques on electronic health records (EHRs) from memory clinic patients to identify ADRD subtypes. Methods: We used pretrained embeddings of non-ADRD diagnosis codes (International Classification of Diseases, Ninth Revision) and large language model (LLM)?derived embeddings of clinical notes from patient EHRs. Hierarchical clustering of these embeddings was used to identify ADRD subtypes. Clusters were characterized regarding their demographic and clinical features. Results: We analyzed a cohort of 3454 patients with ADRD from a memory clinic at Massachusetts General Hospital, each with a specialist diagnosis. Clustering pretrained embeddings of the non-ADRD diagnosis codes in patient EHRs revealed the following 3 patient subtypes: one with skin conditions, another with psychiatric disorders and an earlier age of onset, and a third with diabetes complications. Similarly, using LLM-derived embeddings of clinical notes, we identified 3 subtypes of patients as follows: one with psychiatric manifestations and higher prevalence of female participants (prevalence ratio: 1.59), another with cardiovascular and motor problems and higher prevalence of male participants (prevalence ratio: 1.75), and a third one with geriatric health disorders. Notably, we observed significant overlap between clusters from both data modalities (?24=89.4; P<.001). Conclusions: By integrating International Classification of Diseases, Ninth Revision codes and LLM-derived embeddings, our analysis delineated 2 distinct ADRD subtypes with sex-specific comorbid and clinical presentations, offering insights for potential precision medicine approaches. UR - https://aging.jmir.org/2025/1/e65178 UR - http://dx.doi.org/10.2196/65178 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/65178 ER - TY - JOUR AU - Loh, Rong De AU - Hill, D. Elliot AU - Liu, Nan AU - Dawson, Geraldine AU - Engelhard, M. Matthew PY - 2025/3/27 TI - Limitations of Binary Classification for Long-Horizon Diagnosis Prediction and Advantages of a Discrete-Time Time-to-Event Approach: Empirical Analysis JO - JMIR AI SP - e62985 VL - 4 KW - machine learning KW - artificial intelligence KW - deep learning KW - predictive models KW - practical models KW - early detection KW - electronic health records KW - right-censoring KW - survival analysis KW - distributional shifts N2 - Background: A major challenge in using electronic health records (EHR) is the inconsistency of patient follow-up, resulting in right-censored outcomes. This becomes particularly problematic in long-horizon event predictions, such as autism and attention-deficit/hyperactivity disorder (ADHD) diagnoses, where a significant number of patients are lost to follow-up before the outcome can be observed. Consequently, fully supervised methods such as binary classification (BC), which are trained to predict observed diagnoses, are substantially affected by the probability of sufficient follow-up, leading to biased results. Objective: This empirical analysis aims to characterize BC?s inherent limitations for long-horizon diagnosis prediction from EHR; and quantify the benefits of a specific time-to-event (TTE) approach, the discrete-time neural network (DTNN). Methods: Records within the Duke University Health System EHR were analyzed, extracting features such as ICD-10 (International Classification of Diseases, Tenth Revision) diagnosis codes, medications, laboratories, and procedures. We compared a DTNN to 3 BC approaches and a deep Cox proportional hazards model across 4 clinical conditions to examine distributional patterns across various subgroups. Time-varying area under the receiving operating characteristic curve (AUCt) and time-varying average precision (APt) were our primary evaluation metrics. Results: TTE models consistently had comparable or higher AUCt and APt than BC for all conditions. At clinically relevant operating time points, the area under the receiving operating characteristic curve (AUC) values for DTNNYOB?2020 (year-of-birth) and DCPHYOB?2020 (deep Cox proportional hazard) were 0.70 (95% CI 0.66?0.77) and 0.72 (95% CI 0.66?0.78) at t=5 for autism, 0.72 (95% CI 0.65?0.76) and 0.68 (95% CI 0.62?0.74) at t=7 for ADHD, 0.72 (95% CI 0.70?0.75) and 0.71 (95% CI 0.69?0.74) at t=1 for recurrent otitis media, and 0.74 (95% CI 0.68?0.82) and 0.71 (95% CI 0.63?0.77) at t=1 for food allergy, compared to 0.6 (95% CI 0.55?0.66), 0.47 (95% CI 0.40?0.54), 0.73 (95% CI 0.70?0.75), and 0.77 (95% CI 0.71?0.82) for BCYOB?2020, respectively. The probabilities predicted by BC models were positively correlated with censoring times, particularly for autism and ADHD prediction. Filtering strategies based on YOB or length of follow-up only partially corrected these biases. In subgroup analyses, only DTNN predicted diagnosis probabilities that accurately reflect actual clinical prevalence and temporal trends. Conclusions: BC models substantially underpredicted diagnosis likelihood and inappropriately assigned lower probability scores to individuals with earlier censoring. Common filtering strategies did not adequately address this limitation. TTE approaches, particularly DTNN, effectively mitigated bias from the censoring distribution, resulting in superior discrimination and calibration performance and more accurate prediction of clinical prevalence. Machine learning practitioners should recognize the limitations of BC for long-horizon diagnosis prediction and adopt TTE approaches. The DTNN in particular is well-suited to mitigate the effects of right-censoring and maximize prediction performance in this setting. UR - https://ai.jmir.org/2025/1/e62985 UR - http://dx.doi.org/10.2196/62985 ID - info:doi/10.2196/62985 ER - TY - JOUR AU - Roshani, Amin Mohammad AU - Zhou, Xiangyu AU - Qiang, Yao AU - Suresh, Srinivasan AU - Hicks, Steven AU - Sethuraman, Usha AU - Zhu, Dongxiao PY - 2025/3/27 TI - Generative Large Language Model?Powered Conversational AI App for Personalized Risk Assessment: Case Study in COVID-19 JO - JMIR AI SP - e67363 VL - 4 KW - personalized risk assessment KW - large language model KW - conversational AI KW - artificial intelligence KW - COVID-19 N2 - Background: Large language models (LLMs) have demonstrated powerful capabilities in natural language tasks and are increasingly being integrated into health care for tasks like disease risk assessment. Traditional machine learning methods rely on structured data and coding, limiting their flexibility in dynamic clinical environments. This study presents a novel approach to disease risk assessment using generative LLMs through conversational artificial intelligence (AI), eliminating the need for programming. Objective: This study evaluates the use of pretrained generative LLMs, including LLaMA2-7b and Flan-T5-xl, for COVID-19 severity prediction with the goal of enabling a real-time, no-code, risk assessment solution through chatbot-based, question-answering interactions. To contextualize their performance, we compare LLMs with traditional machine learning classifiers, such as logistic regression, extreme gradient boosting (XGBoost), and random forest, which rely on tabular data. Methods: We fine-tuned LLMs using few-shot natural language examples from a dataset of 393 pediatric patients, developing a mobile app that integrates these models to provide real-time, no-code, COVID-19 severity risk assessment through clinician-patient interaction. The LLMs were compared with traditional classifiers across different experimental settings, using the area under the curve (AUC) as the primary evaluation metric. Feature importance derived from LLM attention layers was also analyzed to enhance interpretability. Results: Generative LLMs demonstrated strong performance in low-data settings. In zero-shot scenarios, the T0-3b-T model achieved an AUC of 0.75, while other LLMs, such as T0pp(8bit)-T and Flan-T5-xl-T, reached 0.67 and 0.69, respectively. At 2-shot settings, logistic regression and random forest achieved an AUC of 0.57, while Flan-T5-xl-T and T0-3b-T obtained 0.69 and 0.65, respectively. By 32-shot settings, Flan-T5-xl-T reached 0.70, similar to logistic regression (0.69) and random forest (0.68), while XGBoost improved to 0.65. These results illustrate the differences in how generative LLMs and traditional models handle the increasing data availability. LLMs perform well in low-data scenarios, whereas traditional models rely more on structured tabular data and labeled training examples. Furthermore, the mobile app provides real-time, COVID-19 severity assessments and personalized insights through attention-based feature importance, adding value to the clinical interpretation of the results. Conclusions: Generative LLMs provide a robust alternative to traditional classifiers, particularly in scenarios with limited labeled data. Their ability to handle unstructured inputs and deliver personalized, real-time assessments without coding makes them highly adaptable to clinical settings. This study underscores the potential of LLM-powered conversational artificial intelligence (AI) in health care and encourages further exploration of its use for real-time, disease risk assessment and decision-making support. UR - https://ai.jmir.org/2025/1/e67363 UR - http://dx.doi.org/10.2196/67363 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/67363 ER - TY - JOUR AU - Li, Jiajia AU - Wang, Zikai AU - Yu, Longxuan AU - Liu, Hui AU - Song, Haitao PY - 2025/3/19 TI - Synthetic Data-Driven Approaches for Chinese Medical Abstract Sentence Classification: Computational Study JO - JMIR Form Res SP - e54803 VL - 9 KW - medical abstract sentence classification KW - large language models KW - synthetic datasets KW - deep learning KW - Chinese medical KW - dataset KW - traditional Chinese medicine KW - global medical research KW - algorithm KW - robustness KW - efficiency KW - accuracy N2 - Background: Medical abstract sentence classification is crucial for enhancing medical database searches, literature reviews, and generating new abstracts. However, Chinese medical abstract classification research is hindered by a lack of suitable datasets. Given the vastness of Chinese medical literature and the unique value of traditional Chinese medicine, precise classification of these abstracts is vital for advancing global medical research. Objective: This study aims to address the data scarcity issue by generating a large volume of labeled Chinese abstract sentences without manual annotation, thereby creating new training datasets. Additionally, we seek to develop more accurate text classification algorithms to improve the precision of Chinese medical abstract classification. Methods: We developed 3 training datasets (dataset #1, dataset #2, and dataset #3) and a test dataset to evaluate our model. Dataset #1 contains 15,000 abstract sentences translated from the PubMed dataset into Chinese. Datasets #2 and #3, each with 15,000 sentences, were generated using GPT-3.5 from 40,000 Chinese medical abstracts in the CSL database. Dataset #2 used titles and keywords for pseudolabeling, while dataset #3 aligned abstracts with category labels. The test dataset includes 87,000 sentences from 20,000 abstracts. We used SBERT embeddings for deeper semantic analysis and evaluated our model using clustering (SBERT-DocSCAN) and supervised methods (SBERT-MEC). Extensive ablation studies and feature analyses were conducted to validate the model?s effectiveness and robustness. Results: Our experiments involved training both clustering and supervised models on the 3 datasets, followed by comprehensive evaluation using the test dataset. The outcomes demonstrated that our models outperformed the baseline metrics. Specifically, when trained on dataset #1, the SBERT-DocSCAN model registered an impressive accuracy and F1-score of 89.85% on the test dataset. Concurrently, the SBERT-MEC algorithm exhibited comparable performance with an accuracy of 89.38% and an identical F1-score. Training on dataset #2 yielded similarly positive results for the SBERT-DocSCAN model, achieving an accuracy and F1-score of 89.83%, while the SBERT-MEC algorithm recorded an accuracy of 86.73% and an F1-score of 86.51%. Notably, training with dataset #3 allowed the SBERT-DocSCAN model to attain the best with an accuracy and F1-score of 91.30%, whereas the SBERT-MEC algorithm also showed robust performance, obtaining an accuracy of 90.39% and an F1-score of 90.35%. Ablation analysis highlighted the critical role of integrated features and methodologies in improving classification efficiency. Conclusions: Our approach addresses the challenge of limited datasets for Chinese medical abstract classification by generating novel datasets. The deployment of SBERT-DocSCAN and SBERT-MEC models significantly enhances the precision of classifying Chinese medical abstracts, even when using synthetic datasets with pseudolabels. UR - https://formative.jmir.org/2025/1/e54803 UR - http://dx.doi.org/10.2196/54803 ID - info:doi/10.2196/54803 ER - TY - JOUR AU - Benaïche, Alexandre AU - Billaut-Laden, Ingrid AU - Randriamihaja, Herivelo AU - Bertocchio, Jean-Philippe PY - 2025/3/10 TI - Assessment of the Efficiency of a ChatGPT-Based Tool, MyGenAssist, in an Industry Pharmacovigilance Department for Case Documentation: Cross-Over Study JO - J Med Internet Res SP - e65651 VL - 27 KW - MyGenAssist KW - large language model KW - artificial intelligence KW - ChatGPT KW - pharmacovigilance KW - efficiency N2 - Background: At the end of 2023, Bayer AG launched its own internal large language model (LLM), MyGenAssist, based on ChatGPT technology to overcome data privacy concerns. It may offer the possibility to decrease their harshness and save time spent on repetitive and recurrent tasks that could then be dedicated to activities with higher added value. Although there is a current worldwide reflection on whether artificial intelligence should be integrated into pharmacovigilance, medical literature does not provide enough data concerning LLMs and their daily applications in such a setting. Here, we studied how this tool could improve the case documentation process, which is a duty for authorization holders as per European and French good vigilance practices. Objective: The aim of the study is to test whether the use of an LLM could improve the pharmacovigilance documentation process. Methods: MyGenAssist was trained to draft templates for case documentation letters meant to be sent to the reporters. Information provided within the template changes depending on the case: such data come from a table sent to the LLM. We then measured the time spent on each case for a period of 4 months (2 months before using the tool and 2 months after its implementation). A multiple linear regression model was created with the time spent on each case as the explained variable, and all parameters that could influence this time were included as explanatory variables (use of MyGenAssist, type of recipient, number of questions, and user). To test if the use of this tool impacts the process, we compared the recipients? response rates with and without the use of MyGenAssist. Results: An average of 23.3% (95% CI 13.8%-32.8%) of time saving was made thanks to MyGenAssist (P<.001; adjusted R2=0.286) on each case, which could represent an average of 10.7 (SD 3.6) working days saved each year. The answer rate was not modified by the use of MyGenAssist (20/48, 42% vs 27/74, 36%; P=.57) whether the recipient was a physician or a patient. No significant difference was found regarding the time spent by the recipient to answer (mean 2.20, SD 3.27 days vs mean 2.65, SD 3.30 days after the last attempt of contact; P=.64). The implementation of MyGenAssist for this activity only required a 2-hour training session for the pharmacovigilance team. Conclusions: Our study is the first to show that a ChatGPT-based tool can improve the efficiency of a good practice activity without needing a long training session for the affected workforce. These first encouraging results could be an incentive for the implementation of LLMs in other processes. UR - https://www.jmir.org/2025/1/e65651 UR - http://dx.doi.org/10.2196/65651 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/65651 ER - TY - JOUR AU - Malik, Salma AU - Dorothea, Pana Zoi AU - Argyropoulos, D. Christos AU - Themistocleous, Sophia AU - Macken, J. Alan AU - Valdenmaiier, Olena AU - Scheckenbach, Frank AU - Bardach, Elena AU - Pfeiffer, Andrea AU - Loens, Katherine AU - Ochando, Cano Jordi AU - Cornely, A. Oliver AU - Demotes-Mainard, Jacques AU - Contrino, Sergio AU - Felder, Gerd PY - 2025/3/7 TI - Data Interoperability in COVID-19 Vaccine Trials: Methodological Approach in the VACCELERATE Project JO - JMIR Med Inform SP - e65590 VL - 13 KW - interoperability KW - metadata KW - data management KW - clinical trials KW - protocol KW - harmonization KW - adult KW - pediatric KW - systems KW - standards N2 - Background: Data standards are not only key to making data processing efficient but also fundamental to ensuring data interoperability. When clinical trial data are structured according to international standards, they become significantly easier to analyze, reducing the efforts required for data cleaning, preprocessing, and secondary use. A common language and a shared set of expectations facilitate interoperability between systems and devices. Objective: The main objectives of this study were to identify commonalities and differences in clinical trial metadata, protocols, and data collection systems/items within the VACCELERATE project. Methods: To assess the degree of interoperability achieved in the project and suggest methodological improvements, interoperable points were identified based on the core outcome areas?immunogenicity, safety, and efficacy (clinical/physiological). These points were emphasized in the development of the master protocol template and were manually compared in the following ways: (1) summaries, objectives, and end points in the protocols of 3 VACCELERATE clinical trials (EU-COVAT-1_AGED, EU-COVAT-2_BOOSTAVAC, and EU-COVPT-1_CoVacc) against the master protocol template; (2) metadata of all 3 clinical trials; and (3) evaluations from a questionnaire survey regarding differences in data management systems and structures that enabled data exchange within the VACCELERATE network. Results: The noncommonalities identified in the protocols and metadata were attributed to differences in populations, variations in protocol design, and vaccination patterns. The detailed metadata released for all 3 vaccine trials were clearly structured using internal standards, terminology, and the general approach of Clinical Data Acquisition Standards Harmonisation (CDASH) for data collection (eg, on electronic case report forms). VACCELERATE benefited significantly from the selection of the Clinical Trials Centre Cologne as the sole data management provider. With system database development coordinated by a single individual and no need for coordination among different trial units, a high degree of uniformity was achieved automatically. The harmonized transfer of data to all sites, using well-established methods, enabled quick exchanges and provided a relatively secure means of data transfer. Conclusions: This study demonstrated that using master protocols can significantly enhance trial operational efficiency and data interoperability, provided that similar infrastructure and data management procedures are adopted across multiple trials. To further improve data interoperability and facilitate interpretation and analysis, shared data should be structured, described, formatted, and stored using widely recognized data and metadata standards. Trial Registration: EudraCT 2021-004526-29; https://www.clinicaltrialsregister.eu/ctr-search/trial/2021-004526-29/DE/; 2021-004889-35; https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2021-004889-35; and 2021-004526-29; https://www.clinicaltrialsregister.eu/ctr-search/search?query=eudract_number:2021-004526-29 UR - https://medinform.jmir.org/2025/1/e65590 UR - http://dx.doi.org/10.2196/65590 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/65590 ER - TY - JOUR AU - Rajaram, Akshay AU - Judd, Michael AU - Barber, David PY - 2025/3/7 TI - Deep Learning Models to Predict Diagnostic and Billing Codes Following Visits to a Family Medicine Practice: Development and Validation Study JO - JMIR AI SP - e64279 VL - 4 KW - machine learning KW - ML KW - artificial intelligence KW - algorithm KW - predictive model KW - predictive analytics KW - predictive system KW - family medicine KW - primary care KW - family doctor KW - family physician KW - income KW - billing code KW - electronic notes KW - electronic health record KW - electronic medical record KW - EMR KW - patient record KW - health record KW - personal health record N2 - Background: Despite significant time spent on billing, family physicians routinely make errors and miss billing opportunities. In other disciplines, machine learning models have predicted Current Procedural Terminology codes with high accuracy. Objective: Our objective was to derive machine learning models capable of predicting diagnostic and billing codes from notes recorded in the electronic medical record. Methods: We conducted a retrospective algorithm development and validation study involving an academic family medicine practice. Visits between July 1, 2015, and June 30, 2020, containing a physician-authored note and an invoice in the electronic medical record were eligible for inclusion. We trained 2 deep learning models and compared their predictions to codes submitted for reimbursement. We calculated accuracy, recall, precision, F1-score, and area under the receiver operating characteristic curve. Results: Of the 245,045 visits eligible for inclusion, 198,802 (81%) were included in model development. Accuracy was 99.8% and 99.5% for the diagnostic and billing code models, respectively. Recall was 49.4% and 70.3% for the diagnostic and billing code models, respectively. Precision was 55.3% and 76.7% for the diagnostic and billing code models, respectively. The area under the receiver operating characteristic curve was 0.983 for the diagnostic code model and 0.993 for the billing code model. Conclusions: We developed models capable of predicting diagnostic and billing codes from electronic notes following visits to a family medicine practice. The billing code model outperformed the diagnostic code model in terms of recall and precision, likely due to fewer codes being predicted. Work is underway to further enhance model performance and assess the generalizability of these models to other family medicine practices. UR - https://ai.jmir.org/2025/1/e64279 UR - http://dx.doi.org/10.2196/64279 ID - info:doi/10.2196/64279 ER - TY - JOUR AU - Ohno, Yukiko AU - Aomori, Tohru AU - Nishiyama, Tomohiro AU - Kato, Riri AU - Fujiki, Reina AU - Ishikawa, Haruki AU - Kiyomiya, Keisuke AU - Isawa, Minae AU - Mochizuki, Mayumi AU - Aramaki, Eiji AU - Ohtani, Hisakazu PY - 2025/3/4 TI - Performance Improvement of a Natural Language Processing Tool for Extracting Patient Narratives Related to Medical States From Japanese Pharmaceutical Care Records by Increasing the Amount of Training Data: Natural Language Processing Analysis and Validation Study JO - JMIR Med Inform SP - e68863 VL - 13 KW - natural language processing KW - NLP KW - named entity recognition KW - NER KW - deep learning KW - pharmaceutical care record KW - electronic medical record KW - EMR KW - Japanese N2 - Background: Patients? oral expressions serve as valuable sources of clinical information to improve pharmacotherapy. Natural language processing (NLP) is a useful approach for analyzing unstructured text data, such as patient narratives. However, few studies have focused on using NLP for narratives in the Japanese language. Objective: We aimed to develop a high-performance NLP system for extracting clinical information from patient narratives by examining the performance progression with a gradual increase in the amount of training data. Methods: We used subjective texts from the pharmaceutical care records of Keio University Hospital from April 1, 2018, to March 31, 2019, comprising 12,004 records from 6559 cases. After preprocessing, we annotated diseases and symptoms within the texts. We then trained and evaluated a deep learning model (bidirectional encoder representations from transformers combined with a conditional random field [BERT-CRF]) through 10-fold cross-validation. The annotated data were divided into 10 subsets, and the amount of training data was progressively increased over 10 steps. We also analyzed the causes of errors. Finally, we applied the developed system to the analysis of case report texts to evaluate its usability for texts from other sources. Results: The F1-score of the system improved from 0.67 to 0.82 as the amount of training data increased from 1200 to 12,004 records. The F1-score reached 0.78 with 3600 records and was largely similar thereafter. As performance improved, errors from incorrect extractions decreased significantly, which resulted in an increase in precision. For case reports, the F1-score also increased from 0.34 to 0.41 as the training dataset expanded from 1200 to 12,004 records. Performance was lower for extracting symptoms from case report texts compared with pharmaceutical care records, suggesting that this system is more specialized for analyzing subjective data from pharmaceutical care records. Conclusions: We successfully developed a high-performance system specialized in analyzing subjective data from pharmaceutical care records by training a large dataset, with near-complete saturation of system performance with about 3600 training records. This system will be useful for monitoring symptoms, offering benefits for both clinical practice and research. UR - https://medinform.jmir.org/2025/1/e68863 UR - http://dx.doi.org/10.2196/68863 UR - http://www.ncbi.nlm.nih.gov/pubmed/40053805 ID - info:doi/10.2196/68863 ER - TY - JOUR AU - Ohlsen, Tessa AU - Hofer, Viola AU - Ingenerf, Josef PY - 2025/2/28 TI - A Validation Tool (VaPCE) for Postcoordinated SNOMED CT Expressions: Development and Usability Study JO - JMIR Med Inform SP - e67984 VL - 13 KW - SNOMED CT KW - PCE KW - postcoordination KW - FHIR KW - validation KW - postcoordinated expression KW - Fast Healthcare Interoperability Resource N2 - Background: The digitalization of health care has increased the demand for efficient data exchange, emphasizing semantic interoperability. SNOMED Clinical Terms (SNOMED CT), a comprehensive terminology with over 360,000 medical concepts, supports this need. However, it cannot cover all medical scenarios, particularly in complex cases. To address this, SNOMED CT allows postcoordination, where users combine precoordinated concepts with new expressions. Despite SNOMED CT?s potential, the creation and validation of postcoordinated expressions (PCEs) remain challenging due to complex syntactic and semantic rules. Objective: This work aims to develop a tool that validates postcoordinated SNOMED CT expressions, focusing on providing users with detailed, automated correction instructions for syntactic and semantic errors. The goal is not just validation, but also offering user-friendly, actionable suggestions for improving PCEs. Methods: A tool was created using the Fast Healthcare Interoperability Resource (FHIR) service $validate-code and the terminology server Ontoserver to check the correctness of PCEs. When errors are detected, the tool processes the SNOMED CT Concept Model in JSON format and applies predefined error categories. For each error type, specific correction suggestions are generated and displayed to users. The key added value of the tool is in generating specific correction suggestions for each identified error, which are displayed to the users. The tool was integrated into a web application, where users can validate individual PCEs or bulk-upload files. The tool was tested with real existing PCEs, which were used as input and validated. In the event of errors, appropriate error messages were generated as output. Results: In the validation of 136 PCEs from 304 FHIR Questionnaires, 18 (13.2%) PCEs were invalid, with the most common errors being invalid attribute values. Additionally, 868 OncoTree codes were evaluated, resulting in 161 (20.9%) PCEs containing inactive concepts, which were successfully replaced with valid alternatives. A user survey reflects a favorable evaluation of the tool?s functionality. Participants found the error categorization and correction suggestions to be precise, offering clear guidance for addressing issues. However, there is potential for enhancement, particularly regarding the level of detail in the error messages. Conclusions: The validation tool significantly improves the accuracy of postcoordinated SNOMED CT expressions by not only identifying errors but also offering detailed correction instructions. This approach supports health care professionals in ensuring that their PCEs are syntactically and semantically valid, enhancing data quality and interoperability across systems. UR - https://medinform.jmir.org/2025/1/e67984 UR - http://dx.doi.org/10.2196/67984 ID - info:doi/10.2196/67984 ER - TY - JOUR AU - Scheider, Simon AU - Mallick, Kamal Mostafa PY - 2025/2/18 TI - Exploring Metadata Catalogs in Health Care Data Ecosystems: Taxonomy Development Study JO - JMIR Form Res SP - e63396 VL - 9 KW - data catalogs KW - data ecosystems KW - findability, accessibility, interoperability, and reusability KW - FAIR KW - health care KW - metadata KW - taxonomy N2 - Background: In the European health care industry, recent years have seen increasing investments in data ecosystems to ?FAIRify? and capitalize the ever-rising amount of health data. Within such networks, health metadata catalogs (HMDCs) assume a key function as they enable data allocation, sharing, and use practices. By design, HMDCs orchestrate health information for the purpose of findability, accessibility, interoperability, and reusability (FAIR). However, despite various European initiatives pushing health care data ecosystems forward, actionable design knowledge about HMDCs is scarce. This impedes both their effective development in practice and their scientific exploration, causing huge unused innovation potential of health data. Objective: This study aims to explore the structural design elements of HMDCs, classifying them alongside empirically reasonable dimensions and characteristics. In doing so, the development of HMDCs in practice is facilitated while also closing a crucial gap in theory (ie, the literature about actionable HMDC design knowledge). Methods: We applied a rigorous methodology for taxonomy building following well-known and established guidelines from the domain of information systems. Within this methodological framework, inductive and deductive research methods were applied to iteratively design and evaluate the evolving set of HMDC dimensions and characteristics. Specifically, a systematic literature review was conducted to identify and analyze 38 articles, while a multicase study was conducted to examine 17 HMDCs from practice. These findings were evaluated and refined in 2 extensive focus group sessions by 7 interdisciplinary experts with deep knowledge about HMDCs. Results: The artifact generated by the study is an iteratively conceptualized and empirically grounded taxonomy with elaborate explanations. It proposes 20 dimensions encompassing 101 characteristics alongside which FAIR HMDCs can be structured and classified. The taxonomy describes basic design characteristics that need to be considered to implement FAIR HMDCs effectively. A major finding was that a particular focus in developing HMDCs is on the design of their published dataset offerings (ie, their metadata assets) as well as on data security and governance. The taxonomy is evaluated against the background of 4 use cases, which were cocreated with experts. These illustrative scenarios add depth and context to the taxonomy as they underline its relevance and applicability in real-world settings. Conclusions: The findings contribute fundamental, yet actionable, design knowledge for building HMDCs in European health care data ecosystems. They provide guidance for health care practitioners, while allowing both scientists and policy makers to navigate through this evolving research field and anchor their work. Therefore, this study closes the research gap outlined earlier, which has prevailed in theory and practice. UR - https://formative.jmir.org/2025/1/e63396 UR - http://dx.doi.org/10.2196/63396 UR - http://www.ncbi.nlm.nih.gov/pubmed/39964739 ID - info:doi/10.2196/63396 ER - TY - JOUR AU - Lu, An-Tai AU - Liou, Chong-Sin AU - Lai, Chia-Hsin AU - Shian, Bo-Tsz AU - Li, Ming-Ta AU - Sun, Chih-Yen AU - Kao, Hao-Yun AU - Dai, Hong-Jie AU - Tsai, Ming-Ju PY - 2025/2/12 TI - Application of Clinical Department?Specific AI-Assisted Coding Using Taiwan Diagnosis-Related Groups: Retrospective Validation Study JO - JMIR Hum Factors SP - e59961 VL - 12 KW - diagnosis-related group KW - artificial intelligence coding KW - International Classification of Diseases, Tenth Revision, Clinical Modification KW - ICD-10-CM KW - coding professionals N2 - Background: The accuracy of the ICD-10-CM (International Classification of Diseases, Tenth Revision, Clinical Modification) procedure coding system (PCS) is crucial for generating correct Taiwan diagnosis-related groups (DRGs), as coding errors can lead to financial losses for hospitals. Objective: The study aimed to determine the consistency between an artificial intelligence (AI)-assisted coding module and manual coding, as well as to identify clinical specialties suitable for implementing the developed AI-assisted coding module. Methods: This study examined the AI-assisted coding module from the perspective of health care professionals. The research period started in February 2023. The study excluded cases outside of Taiwan DRGs, those with incomplete medical records, and cases with Taiwan DRG disposals ICD-10 (International Statistical Classification of Diseases, Tenth Revision) PCS. Data collection was conducted through retrospective medical record review. The AI-assisted module was constructed using a hierarchical attention network. The verification of the Taiwan DRGs results from the AI-assisted coding model focused on the major diagnostic categories (MDCs). Statistical computations were conducted using SPSS version 19. Research variables consisted of categorical variables represented by MDC, and continuous variables were represented by the relative weight of Taiwan DRGs. Results: A total of 2632 discharge records meeting the research criteria were collected from February to April 2023. In terms of inferential statistics, ? statistics were used for MDC analysis. The infectious and parasitic diseases MDC, as well as the respiratory diseases MDC had ? values exceeding 0.8. Clinical inpatient specialties were statistically analyzed using the Wilcoxon signed rank test. There was not a difference in coding results between the 23 clinical departments, such as the Division of Cardiology, the Division of Nephrology, and the Department of Urology. Conclusions: For human coders, with the assistance of the ICD-10-CM AI-assisted coding system, work time is reduced. Additionally, strengthening knowledge in clinical documentation enables human coders to maximize their role. This positions them to become clinical documentation experts, preparing them for further career development. Future research will apply the same method to validate the ICD-10 AI-assisted coding module. UR - https://humanfactors.jmir.org/2025/1/e59961 UR - http://dx.doi.org/10.2196/59961 ID - info:doi/10.2196/59961 ER - TY - JOUR AU - Puts, Sander AU - Zegers, L. Catharina M. AU - Dekker, Andre AU - Bermejo, Iñigo PY - 2025/2/11 TI - Developing an ICD-10 Coding Assistant: Pilot Study Using RoBERTa and GPT-4 for Term Extraction and Description-Based Code Selection JO - JMIR Form Res SP - e60095 VL - 9 KW - International Classification of Diseases KW - ICD-10 KW - computer-assisted-coding KW - GPT-4 KW - coding KW - term extraction KW - code analysis KW - computer assisted coding KW - transformer model KW - artificial intelligence KW - AI automation KW - retrieval-augmented generation KW - RAG KW - large language model KW - LLM KW - Bidirectional Encoder Representations from Transformers KW - Robustly Optimized BERT Pretraining Approach KW - RoBERTa KW - named entity recognition KW - NER N2 - Background: The International Classification of Diseases (ICD), developed by the World Health Organization, standardizes health condition coding to support health care policy, research, and billing, but artificial intelligence automation, while promising, still underperforms compared with human accuracy and lacks the explainability needed for adoption in medical settings. Objective: The potential of large language models for assisting medical coders in the ICD-10 coding was explored through the development of a computer-assisted coding system. This study aimed to augment human coding by initially identifying lead terms and using retrieval-augmented generation (RAG)?based methods for computer-assisted coding enhancement. Methods: The explainability dataset from the CodiEsp challenge (CodiEsp-X) was used, featuring 1000 Spanish clinical cases annotated with ICD-10 codes. A new dataset, CodiEsp-X-lead, was generated using GPT-4 to replace full-textual evidence annotations with lead term annotations. A Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach transformer model was fine-tuned for named entity recognition to extract lead terms. GPT-4 was subsequently employed to generate code descriptions from the extracted textual evidence. Using a RAG approach, ICD codes were assigned to the lead terms by querying a vector database of ICD code descriptions with OpenAI?s text-embedding-ada-002 model. Results: The fine-tuned Robustly Optimized BERT Pretraining Approach achieved an overall F1-score of 0.80 for ICD lead term extraction on the new CodiEsp-X-lead dataset. GPT-4-generated code descriptions reduced retrieval failures in the RAG approach by approximately 5% for both diagnoses and procedures. However, the overall explainability F1-score for the CodiEsp-X task was limited to 0.305, significantly lower than the state-of-the-art F1-score of 0.633. The diminished performance was partly due to the reliance on code descriptions, as some ICD codes lacked descriptions, and the approach did not fully align with the medical coder?s workflow. Conclusions: While lead term extraction showed promising results, the subsequent RAG-based code assignment using GPT-4 and code descriptions was less effective. Future research should focus on refining the approach to more closely mimic the medical coder?s workflow, potentially integrating the alphabetic index and official coding guidelines, rather than relying solely on code descriptions. This alignment may enhance system accuracy and better support medical coders in practice. UR - https://formative.jmir.org/2025/1/e60095 UR - http://dx.doi.org/10.2196/60095 ID - info:doi/10.2196/60095 ER - TY - JOUR AU - Rosner, Benjamin AU - Horridge, Matthew AU - Austria, Guillen AU - Lee, Tiffany AU - Auerbach, Andrew PY - 2025/2/6 TI - An Ontology for Digital Medicine Outcomes: Development of the Digital Medicine Outcomes Value Set (DOVeS) JO - JMIR Med Inform SP - e67589 VL - 13 KW - digital health KW - digital medicine KW - digital therapeutics KW - ontology KW - medical informatics KW - value set KW - development KW - digital health tool KW - DHT KW - health systems KW - digital medicine outcomes value set KW - prototype KW - users N2 - Background: Over the last 10-15 years, US health care and the practice of medicine itself have been transformed by a proliferation of digital medicine and digital therapeutic products (collectively, digital health tools [DHTs]). While a number of DHT classifications have been proposed to help organize these tools for discovery, retrieval, and comparison by health care organizations seeking to potentially implement them, none have specifically addressed that organizations considering their implementation approach the DHT discovery process with one or more specific outcomes in mind. An outcomes-based DHT ontology could therefore be valuable not only for health systems seeking to evaluate tools that influence certain outcomes, but also for regulators and vendors seeking to ascertain potential substantial equivalence to predicate devices. Objective: This study aimed to develop, with inputs from industry, health care providers, payers, regulatory bodies, and patients through the Accelerated Digital Clinical Ecosystem (ADviCE) consortium, an ontology specific to DHT outcomes, the Digital medicine Outcomes Value Set (DOVeS), and to make this ontology publicly available and free to use. Methods: From a starting point of a 4-generation?deep hierarchical taxonomy developed by ADviCE, we developed DOVeS using the Web Ontology Language through the open-source ontology editor Protégé, and data from 185 vendors who had submitted structured product information to ADviCE. We used a custom, decentralized, collaborative ontology engineering methodology, and were guided by Open Biological and Biomedical Ontologies (OBO) Foundry principles. We incorporated the Mondo Disease Ontology (MONDO) and the Ontology of Adverse Events. After development, DOVeS was field-tested between December 2022 and May 2023 with 40 additional independent vendors previously unfamiliar with ADviCE or DOVeS. As a proof of concept, we subsequently developed a prototype DHT Application Finder leveraging DOVeS to enable a user to query for DHT products based on specific outcomes of interest. Results: In its current state, DOVeS contains 42,320 and 9481 native axioms and distinct classes, respectively. These numbers are enhanced when taking into account the axioms and classes contributed by MONDO and the Ontology of Adverse Events. Conclusions: DOVeS is publicly available on BioPortal and GitHub, and has a Creative Commons license CC-BY-SA that is intended to encourage stakeholders to modify, adapt, build upon, and distribute it. While no ontology is complete, DOVeS will benefit from a strong and engaged user base to help it grow and evolve in a way that best serves DHT stakeholders and the patients they serve. UR - https://medinform.jmir.org/2025/1/e67589 UR - http://dx.doi.org/10.2196/67589 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/67589 ER - TY - JOUR AU - Yang, Doris AU - Zhou, Doudou AU - Cai, Steven AU - Gan, Ziming AU - Pencina, Michael AU - Avillach, Paul AU - Cai, Tianxi AU - Hong, Chuan PY - 2025/1/22 TI - Robust Automated Harmonization of Heterogeneous Data Through Ensemble Machine Learning: Algorithm Development and Validation Study JO - JMIR Med Inform SP - e54133 VL - 13 KW - ensemble learning KW - semantic learning KW - distribution learning KW - variable harmonization KW - machine learning KW - cardiovascular health study KW - intracohort comparison KW - intercohort comparison KW - gold standard labels N2 - Background: Cohort studies contain rich clinical data across large and diverse patient populations and are a common source of observational data for clinical research. Because large scale cohort studies are both time and resource intensive, one alternative is to harmonize data from existing cohorts through multicohort studies. However, given differences in variable encoding, accurate variable harmonization is difficult. Objective: We propose SONAR (Semantic and Distribution-Based Harmonization) as a method for harmonizing variables across cohort studies to facilitate multicohort studies. Methods: SONAR used semantic learning from variable descriptions and distribution learning from study participant data. Our method learned an embedding vector for each variable and used pairwise cosine similarity to score the similarity between variables. This approach was built off 3 National Institutes of Health cohorts, including the Cardiovascular Health Study, the Multi-Ethnic Study of Atherosclerosis, and the Women?s Health Initiative. We also used gold standard labels to further refine the embeddings in a supervised manner. Results: The method was evaluated using manually curated gold standard labels from the 3 National Institutes of Health cohorts. We evaluated both the intracohort and intercohort variable harmonization performance. The supervised SONAR method outperformed existing benchmark methods for almost all intracohort and intercohort comparisons using area under the curve and top-k accuracy metrics. Notably, SONAR was able to significantly improve harmonization of concepts that were difficult for existing semantic methods to harmonize. Conclusions: SONAR achieves accurate variable harmonization within and between cohort studies by harnessing the complementary strengths of semantic learning and variable distribution learning. UR - https://medinform.jmir.org/2025/1/e54133 UR - http://dx.doi.org/10.2196/54133 ID - info:doi/10.2196/54133 ER - TY - JOUR AU - Li, Xiadong AU - Shu, Qiang AU - Kong, Canhong AU - Wang, Jinhu AU - Li, Gang AU - Fang, Xin AU - Lou, Xiaomin AU - Yu, Gang PY - 2025/1/8 TI - An Intelligent System for Classifying Patient Complaints Using Machine Learning and Natural Language Processing: Development and Validation Study JO - J Med Internet Res SP - e55721 VL - 27 KW - complaint analysis KW - text classification KW - natural language processing KW - NLP KW - machine learning KW - ML KW - patient complaints N2 - Background: Accurate classification of patient complaints is crucial for enhancing patient satisfaction management in health care settings. Traditional manual methods for categorizing complaints often lack efficiency and precision. Thus, there is a growing demand for advanced and automated approaches to streamline the classification process. Objective: This study aimed to develop and validate an intelligent system for automatically classifying patient complaints using machine learning (ML) and natural language processing (NLP) techniques. Methods: An ML-based NLP technology was proposed to extract frequently occurring dissatisfactory words related to departments, staff, and key treatment procedures. A dataset containing 1465 complaint records from 2019 to 2023 was used for training and validation, with an additional 376 complaints from Hangzhou Cancer Hospital serving as an external test set. Complaints were categorized into 4 types?communication problems, diagnosis and treatment issues, management problems, and sense of responsibility concerns. The imbalanced data were balanced using the Synthetic Minority Oversampling Technique (SMOTE) algorithm to ensure equal representation across all categories. A total of 3 ML algorithms (Multifactor Logistic Regression, Multinomial Naive Bayes, and Support Vector Machines [SVM]) were used for model training and validation. The best-performing model was tested using a 5-fold cross-validation on external data. Results: The original dataset consisted of 719, 376, 260, and 86 records for communication problems, diagnosis and treatment issues, management problems, and sense of responsibility concerns, respectively. The Multifactor Logistic Regression and SVM models achieved weighted average accuracies of 0.89 and 0.93 in the training set, and 0.83 and 0.87 in the internal test set, respectively. Ngram-level term frequency?inverse document frequency did not significantly improve classification performance, with only a marginal 1% increase in precision, recall, and F1-score when implementing Ngram-level term frequency?inverse document frequency (n=2) from 0.91 to 0.92. The SVM algorithm performed best in prediction, achieving an average accuracy of 0.91 on the external test set with a 95% CI of 0.87-0.97. Conclusions: The NLP-driven SVM algorithm demonstrates effective classification performance in automatically categorizing patient complaint texts. It showed superior performance in both internal and external test sets for communication and management problems. However, caution is advised when using it for classifying sense of responsibility complaints. This approach holds promises for implementation in medical institutions with high complaint volumes and limited resources for addressing patient feedback. UR - https://www.jmir.org/2025/1/e55721 UR - http://dx.doi.org/10.2196/55721 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/55721 ER - TY - JOUR AU - Dimitsaki, Stella AU - Natsiavas, Pantelis AU - Jaulent, Marie-Christine PY - 2024/12/30 TI - Applying AI to Structured Real-World Data for Pharmacovigilance Purposes: Scoping Review JO - J Med Internet Res SP - e57824 VL - 26 KW - pharmacovigilance KW - drug safety KW - artificial intelligence KW - machine learning KW - real-world data KW - scoping review N2 - Background: Artificial intelligence (AI) applied to real-world data (RWD; eg, electronic health care records) has been identified as a potentially promising technical paradigm for the pharmacovigilance field. There are several instances of AI approaches applied to RWD; however, most studies focus on unstructured RWD (conducting natural language processing on various data sources, eg, clinical notes, social media, and blogs). Hence, it is essential to investigate how AI is currently applied to structured RWD in pharmacovigilance and how new approaches could enrich the existing methodology. Objective: This scoping review depicts the emerging use of AI on structured RWD for pharmacovigilance purposes to identify relevant trends and potential research gaps. Methods: The scoping review methodology is based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology. We queried the MEDLINE database through the PubMed search engine. Relevant scientific manuscripts published from January 2010 to January 2024 were retrieved. The included studies were ?mapped? against a set of evaluation criteria, including applied AI approaches, code availability, description of the data preprocessing pipeline, clinical validation of AI models, and implementation of trustworthy AI criteria following the guidelines of the FUTURE (Fairness, Universality, Traceability, Usability, Robustness, and Explainability)-AI initiative. Results: The scoping review ultimately yielded 36 studies. There has been a significant increase in relevant studies after 2019. Most of the articles focused on adverse drug reaction detection procedures (23/36, 64%) for specific adverse effects. Furthermore, a substantial number of studies (34/36, 94%) used nonsymbolic AI approaches, emphasizing classification tasks. Random forest was the most popular machine learning approach identified in this review (17/36, 47%). The most common RWD sources used were electronic health care records (28/36, 78%). Typically, these data were not available in a widely acknowledged data model to facilitate interoperability, and they came from proprietary databases, limiting their availability for reproducing results. On the basis of the evaluation criteria classification, 10% (4/36) of the studies published their code in public registries, 16% (6/36) tested their AI models in clinical environments, and 36% (13/36) provided information about the data preprocessing pipeline. In addition, in terms of trustworthy AI, 89% (32/36) of the studies followed at least half of the trustworthy AI initiative guidelines. Finally, selection and confounding biases were the most common biases in the included studies. Conclusions: AI, along with structured RWD, constitutes a promising line of work for drug safety and pharmacovigilance. However, in terms of AI, some approaches have not been examined extensively in this field (such as explainable AI and causal AI). Moreover, it would be helpful to have a data preprocessing protocol for RWD to support pharmacovigilance processes. Finally, because of personal data sensitivity, evaluation procedures have to be investigated further. UR - https://www.jmir.org/2024/1/e57824 UR - http://dx.doi.org/10.2196/57824 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/57824 ER - TY - JOUR AU - Knight, Jo AU - Chandrabalan, Vardhan Vishnu AU - Emsley, A. Hedley C. PY - 2024/12/24 TI - Visualizing Patient Pathways and Identifying Data Repositories in a UK Neurosciences Center: Exploratory Study JO - JMIR Med Inform SP - e60017 VL - 12 KW - health data KW - business process monitoring notation KW - neurology KW - process monitoring KW - patient pathway KW - clinical pathway KW - patient care KW - EHR KW - electronic health record KW - dataset KW - questionnaire KW - patient data KW - NHS KW - National Health Service N2 - Background: Health and clinical activity data are a vital resource for research, improving patient care and service efficiency. Health care data are inherently complex, and their acquisition, storage, retrieval, and subsequent analysis require a thorough understanding of the clinical pathways underpinning such data. Better use of health care data could lead to improvements in patient care and service delivery. However, this depends on the identification of relevant datasets. Objective: We aimed to demonstrate the application of business process modeling notation (BPMN) to represent clinical pathways at a UK neurosciences center and map the clinical activity to corresponding data flows into electronic health records and other nonstandard data repositories. Methods: We used BPMN to map and visualize a patient journey and the subsequent movement and storage of patient data. After identifying several datasets that were being held outside of the standard applications, we collected information about these datasets using a questionnaire. Results: We identified 13 standard applications where neurology clinical activity was captured as part of the patient?s electronic health record including applications and databases for managing referrals, outpatient activity, laboratory data, imaging data, and clinic letters. We also identified 22 distinct datasets not within standard applications that were created and managed within the neurosciences department, either by individuals or teams. These were being used to deliver direct patient care and included datasets for tracking patient blood results, recording home visits, and tracking triage status. Conclusions: Mapping patient data flows and repositories allowed us to identify areas wherein the current electronic health record does not fulfill the needs of day-to-day patient care. Data that are being stored outside of standard applications represent a potential duplication in the effort and risks being overlooked. Future work should identify unmet data needs to inform correct data capture and centralization within appropriate data architectures. UR - https://medinform.jmir.org/2024/1/e60017 UR - http://dx.doi.org/10.2196/60017 ID - info:doi/10.2196/60017 ER - TY - JOUR AU - Kim, Sanghwan AU - Jang, Sowon AU - Kim, Borham AU - Sunwoo, Leonard AU - Kim, Seok AU - Chung, Jin-Haeng AU - Nam, Sejin AU - Cho, Hyeongmin AU - Lee, Donghyoung AU - Lee, Keehyuck AU - Yoo, Sooyoung PY - 2024/12/20 TI - Automated Pathologic TN Classification Prediction and Rationale Generation From Lung Cancer Surgical Pathology Reports Using a Large Language Model Fine-Tuned With Chain-of-Thought: Algorithm Development and Validation Study JO - JMIR Med Inform SP - e67056 VL - 12 KW - AJCC Cancer Staging Manual 8th edition KW - American Joint Committee on Cancer KW - large language model KW - chain-of-thought KW - rationale KW - lung cancer KW - report analysis KW - AI KW - surgery KW - pathology reports KW - tertiary hospital KW - generative language models KW - efficiency KW - accuracy KW - automated N2 - Background: Traditional rule-based natural language processing approaches in electronic health record systems are effective but are often time-consuming and prone to errors when handling unstructured data. This is primarily due to the substantial manual effort required to parse and extract information from diverse types of documentation. Recent advancements in large language model (LLM) technology have made it possible to automatically interpret medical context and support pathologic staging. However, existing LLMs encounter challenges in rapidly adapting to specialized guideline updates. In this study, we fine-tuned an LLM specifically for lung cancer pathologic staging, enabling it to incorporate the latest guidelines for pathologic TN classification. Objective: This study aims to evaluate the performance of fine-tuned generative language models in automatically inferring pathologic TN classifications and extracting their rationale from lung cancer surgical pathology reports. By addressing the inefficiencies and extensive parsing efforts associated with rule-based methods, this approach seeks to enable rapid and accurate reclassification aligned with the latest cancer staging guidelines. Methods: We conducted a comparative performance evaluation of 6 open-source LLMs for automated TN classification and rationale generation, using 3216 deidentified lung cancer surgical pathology reports based on the American Joint Committee on Cancer (AJCC) Cancer Staging Manual8th edition, collected from a tertiary hospital. The dataset was preprocessed by segmenting each report according to lesion location and morphological diagnosis. Performance was assessed using exact match ratio (EMR) and semantic match ratio (SMR) as evaluation metrics, which measure classification accuracy and the contextual alignment of the generated rationales, respectively. Results: Among the 6 models, the Orca2_13b model achieved the highest performance with an EMR of 0.934 and an SMR of 0.864. The Orca2_7b model also demonstrated strong performance, recording an EMR of 0.914 and an SMR of 0.854. In contrast, the Llama2_7b model achieved an EMR of 0.864 and an SMR of 0.771, while the Llama2_13b model showed an EMR of 0.762 and an SMR of 0.690. The Mistral_7b and Llama3_8b models, on the other hand, showed lower performance, with EMRs of 0.572 and 0.489, and SMRs of 0.377 and 0.456, respectively. Overall, the Orca2 models consistently outperformed the others in both TN stage classification and rationale generation. Conclusions: The generative language model approach presented in this study has the potential to enhance and automate TN classification in complex cancer staging, supporting both clinical practice and oncology data curation. With additional fine-tuning based on cancer-specific guidelines, this approach can be effectively adapted to other cancer types. UR - https://medinform.jmir.org/2024/1/e67056 UR - http://dx.doi.org/10.2196/67056 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/67056 ER - TY - JOUR AU - de Groot, Rowdy AU - van der Graaff, Frank AU - van der Doelen, Daniël AU - Luijten, Michiel AU - De Meyer, Ronald AU - Alrouh, Hekmat AU - van Oers, Hedy AU - Tieskens, Jacintha AU - Zijlmans, Josjan AU - Bartels, Meike AU - Popma, Arne AU - de Keizer, Nicolette AU - Cornet, Ronald AU - Polderman, C. Tinca J. PY - 2024/12/19 TI - Implementing Findable, Accessible, Interoperable, Reusable (FAIR) Principles in Child and Adolescent Mental Health Research: Mixed Methods Approach JO - JMIR Ment Health SP - e59113 VL - 11 KW - FAIR data KW - research data management KW - data interoperability KW - data standardization KW - OMOP CDM KW - implementation KW - health data KW - data quality KW - FAIR principles N2 - Background: The FAIR (Findable, Accessible, Interoperable, Reusable) data principles are a guideline to improve the reusability of data. However, properly implementing these principles is challenging due to a wide range of barriers. Objectives: To further the field of FAIR data, this study aimed to systematically identify barriers regarding implementing the FAIR principles in the area of child and adolescent mental health research, define the most challenging barriers, and provide recommendations for these barriers. Methods: Three sources were used as input to identify barriers: (1) evaluation of the implementation process of the Observational Medical Outcomes Partnership Common Data Model by 3 data managers; (2) interviews with experts on mental health research, reusable health data, and data quality; and (3) a rapid literature review. All barriers were categorized according to type as described previously, the affected FAIR principle, a category to add detail about the origin of the barrier, and whether a barrier was mental health specific. The barriers were assessed and ranked on impact with the data managers using the Delphi method. Results: Thirteen barriers were identified by the data managers, 7 were identified by the experts, and 30 barriers were extracted from the literature. This resulted in 45 unique barriers. The characteristics that were most assigned to the barriers were, respectively, external type (n=32/45; eg, organizational policy preventing the use of required software), tooling category (n=19/45; ie, software and databases), all FAIR principles (n=15/45), and not mental health specific (n=43/45). Consensus on ranking the scores of the barriers was reached after 2 rounds of the Delphi method. The most important recommendations to overcome the barriers are adding a FAIR data steward to the research team, accessible step-by-step guides, and ensuring sustainable funding for the implementation and long-term use of FAIR data. Conclusions: By systematically listing these barriers and providing recommendations, we intend to enhance the awareness of researchers and grant providers that making data FAIR demands specific expertise, available tooling, and proper investments. UR - https://mental.jmir.org/2024/1/e59113 UR - http://dx.doi.org/10.2196/59113 ID - info:doi/10.2196/59113 ER - TY - JOUR AU - Cao, Lang AU - Sun, Jimeng AU - Cross, Adam PY - 2024/12/18 TI - An Automatic and End-to-End System for Rare Disease Knowledge Graph Construction Based on Ontology-Enhanced Large Language Models: Development Study JO - JMIR Med Inform SP - e60665 VL - 12 KW - rare disease KW - clinical informatics KW - LLM KW - natural language processing KW - machine learning KW - artificial intelligence KW - large language models KW - data extraction KW - ontologies KW - knowledge graphs KW - text mining N2 - Background: Rare diseases affect millions worldwide but sometimes face limited research focus individually due to low prevalence. Many rare diseases do not have specific International Classification of Diseases, Ninth Edition (ICD-9) and Tenth Edition (ICD-10), codes and therefore cannot be reliably extracted from granular fields like ?Diagnosis? and ?Problem List? entries, which complicates tasks that require identification of patients with these conditions, including clinical trial recruitment and research efforts. Recent advancements in large language models (LLMs) have shown promise in automating the extraction of medical information, offering the potential to improve medical research, diagnosis, and management. However, most LLMs lack professional medical knowledge, especially concerning specific rare diseases, and cannot effectively manage rare disease data in its various ontological forms, making it unsuitable for these tasks. Objective: Our aim is to create an end-to-end system called automated rare disease mining (AutoRD), which automates the extraction of rare disease?related information from medical text, focusing on entities and their relations to other medical concepts, such as signs and symptoms. AutoRD integrates up-to-date ontologies with other structured knowledge and demonstrates superior performance in rare disease extraction tasks. We conducted various experiments to evaluate AutoRD?s performance, aiming to surpass common LLMs and traditional methods. Methods: AutoRD is a pipeline system that involves data preprocessing, entity extraction, relation extraction, entity calibration, and knowledge graph construction. We implemented this system using GPT-4 and medical knowledge graphs developed from the open-source Human Phenotype and Orphanet ontologies, using techniques such as chain-of-thought reasoning and prompt engineering. We quantitatively evaluated our system?s performance in entity extraction, relation extraction, and knowledge graph construction. The experiment used the well-curated dataset RareDis2023, which contains medical literature focused on rare disease entities and their relations, making it an ideal dataset for training and testing our methodology. Results: On the RareDis2023 dataset, AutoRD achieved an overall entity extraction F1-score of 56.1% and a relation extraction F1-score of 38.6%, marking a 14.4% improvement over the baseline LLM. Notably, the F1-score for rare disease entity extraction reached 83.5%, indicating high precision and recall in identifying rare disease mentions. These results demonstrate the effectiveness of integrating LLMs with medical ontologies in extracting complex rare disease information. Conclusions: AutoRD is an automated end-to-end system for extracting rare disease information from text to build knowledge graphs, addressing critical limitations of existing LLMs by improving identification of these diseases and connecting them to related clinical features. This work underscores the significant potential of LLMs in transforming health care, particularly in the rare disease domain. By leveraging ontology-enhanced LLMs, AutoRD constructs a robust medical knowledge base that incorporates up-to-date rare disease information, facilitating improved identification of patients and resulting in more inclusive research and trial candidacy efforts. UR - https://medinform.jmir.org/2024/1/e60665 UR - http://dx.doi.org/10.2196/60665 ID - info:doi/10.2196/60665 ER - TY - JOUR AU - Albers, W. Charlotte A. AU - Wieland-Jorna, Yvonne AU - de Bruijne, C. Martine AU - Smalbrugge, Martin AU - Joling, J. Karlijn AU - de Boer, E. Marike PY - 2024/12/13 TI - Enhancing Standardized and Structured Recording by Elderly Care Physicians for Reusing Electronic Health Record Data: Interview Study JO - JMIR Med Inform SP - e63710 VL - 12 KW - electronic health records KW - health information interoperability KW - health information exchange KW - reference standards KW - long-term care KW - nursing homes KW - medical records KW - attitude of health personnel KW - qualitative research KW - digital health N2 - Background: Elderly care physicians (ECPs) in nursing homes document patients? health, medical conditions, and the care provided in electronic health records (EHRs). However, much of these health data currently lack structure and standardization, limiting their potential for health information exchange across care providers and reuse for quality improvement, policy development, and scientific research. Enhancing this potential requires insight into the attitudes and behaviors of ECPs toward standardized and structured recording in EHRs. Objective: This study aims to answer why and how ECPs record their findings in EHRs and what factors influence them to record in a standardized and structured manner. The findings will be used to formulate recommendations aimed at enhancing standardized and structured data recording for the reuse of EHR data. Methods: Semistructured interviews were conducted with 13 ECPs working in Dutch nursing homes. We recruited participants through purposive sampling, aiming for diversity in age, gender, health care organization, and use of EHR systems. Interviews continued until we reached data saturation. Analysis was performed using inductive thematic analysis. Results: ECPs primarily use EHRs to document daily patient care, ensure continuity of care, and fulfill their obligation to record specific information for accountability purposes. The EHR serves as a record to justify their actions in the event of a complaint. In addition, some respondents also mentioned recording information for secondary purposes, such as research and quality improvement. Several factors were found to influence standardized and structured recording. At a personal level, it is crucial to experience the added value of standardized and structured recording. At the organizational level, clear internal guidelines and a focus on their implementation can have a substantial impact. At the level of the EHR system, user-friendliness, interoperability, and guidance were most frequently mentioned as being important. At a national level, the alignment of internal guidelines with overarching standards plays a pivotal role in encouraging standardized and structured recording. Conclusions: The results of our study are similar to the findings of previous research in hospital care and general practice. Therefore, long-term care can learn from solutions regarding standardized and structured recording in other health care sectors. The main motives for ECPs to record in EHRs are the daily patient care and ensuring continuity of care. Standardized and structured recording can be improved by aligning the recording method in EHRs with the primary care process. In addition, there are incentives for motivating ECPs to record in a standardized and structured way, mainly at the personal, organizational, EHR system, and national levels. UR - https://medinform.jmir.org/2024/1/e63710 UR - http://dx.doi.org/10.2196/63710 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/63710 ER - TY - JOUR AU - Ralevski, Alexandra AU - Taiyab, Nadaa AU - Nossal, Michael AU - Mico, Lindsay AU - Piekos, Samantha AU - Hadlock, Jennifer PY - 2024/11/19 TI - Using Large Language Models to Abstract Complex Social Determinants of Health From Original and Deidentified Medical Notes: Development and Validation Study JO - J Med Internet Res SP - e63445 VL - 26 KW - housing instability KW - housing insecurity KW - housing KW - machine learning KW - artificial intelligence KW - AI KW - large language model KW - LLM KW - natural language processing KW - NLP KW - electronic health record KW - EHR KW - electronic medical record KW - EMR KW - social determinants of health KW - exposome KW - pregnancy KW - obstetric KW - deidentification N2 - Background: Social determinants of health (SDoH) such as housing insecurity are known to be intricately linked to patients? health status. More efficient methods for abstracting structured data on SDoH can help accelerate the inclusion of exposome variables in biomedical research and support health care systems in identifying patients who could benefit from proactive outreach. Large language models (LLMs) developed from Generative Pre-trained Transformers (GPTs) have shown potential for performing complex abstraction tasks on unstructured clinical notes. Objective: Here, we assess the performance of GPTs on identifying temporal aspects of housing insecurity and compare results between both original and deidentified notes. Methods: We compared the ability of GPT-3.5 and GPT-4 to identify instances of both current and past housing instability, as well as general housing status, from 25,217 notes from 795 pregnant women. Results were compared with manual abstraction, a named entity recognition model, and regular expressions. Results: Compared with GPT-3.5 and the named entity recognition model, GPT-4 had the highest performance and had a much higher recall (0.924) than human abstractors (0.702) in identifying patients experiencing current or past housing instability, although precision was lower (0.850) compared with human abstractors (0.971). GPT-4?s precision improved slightly (0.936 original, 0.939 deidentified) on deidentified versions of the same notes, while recall dropped (0.781 original, 0.704 deidentified). Conclusions: This work demonstrates that while manual abstraction is likely to yield slightly more accurate results overall, LLMs can provide a scalable, cost-effective solution with the advantage of greater recall. This could support semiautomated abstraction, but given the potential risk for harm, human review would be essential before using results for any patient engagement or care decisions. Furthermore, recall was lower when notes were deidentified prior to LLM abstraction. UR - https://www.jmir.org/2024/1/e63445 UR - http://dx.doi.org/10.2196/63445 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/63445 ER - TY - JOUR AU - Bogale, Binyam AU - Vesinurm, Märt AU - Lillrank, Paul AU - Celius, Gulowsen Elisabeth AU - Halvorsrud, Ragnhild PY - 2024/11/15 TI - Visual Modeling Languages in Patient Pathways: Scoping Review JO - Interact J Med Res SP - e55865 VL - 13 KW - patient pathways KW - visual modeling languages KW - business process model and notation KW - BPMN KW - unified modeling language KW - UML KW - domain-specific modeling languages KW - scoping review N2 - Background: Patient pathways (PPs) are presented as a panacea solution to enhance health system functions. It is a complex concept that needs to be described and communicated well. Modeling plays a crucial role in promoting communication, fostering a shared understanding, and streamlining processes. Only a few existing systematic reviews have focused on modeling methods and standardized modeling languages. There remains a gap in consolidated knowledge regarding the use of diverse visual modeling languages. Objective: This scoping review aimed to compile visual modeling languages used to represent PPs, including the justifications and the context in which a modeling language was adopted, adapted, combined, or developed. Methods: After initial experimentation with the keywords used to describe the concepts of PPs and visual modeling languages, we developed a search strategy that was further refined and customized to the major databases identified as topically relevant. In addition, we consulted gray literature and conducted hand searches of the referenced articles. Two reviewers independently screened the articles in 2 stages using preset inclusion criteria, and a third reviewer voted on the discordance. Data charting was done using an iteratively developed form in the Covidence software. Descriptive and thematic summaries were presented following rounds of discussion to produce the final report. Results: Of 1838 articles retrieved after deduplication, 22 satisfied our inclusion criteria. Clinical pathway is the most used phrase to represent the PP concept, and most papers discussed the concept without providing their operational definition. We categorized the visual modeling languages into five categories: (1) general purpose?modeling language (GPML) adopted without major extension or modification, (2) GPML used with formal extension recommendations, (3) combination of 2 or more modeling languages, (4) a developed domain-specific modeling language (DSML), and (5) ontological modeling languages. The justifications for adopting, adapting, combining, and developing visual modeling languages varied accordingly and ranged from versatility, expressiveness, tool support, and extensibility of a language to domain needs, integration, and simplification. Conclusions: Various visual modeling languages were used in PP modeling, each with varying levels of abstraction and granularity. The categorization we made could aid in a better understanding of the complex combination of PP and modeling languages. Standardized GPMLs were used with or without any modifications. The rationale to propose any modification to GPMLs evolved as more evidence was presented following requirement analyses to support domain constructs. DSMLs are infrequently used due to their resource-intensive development, often initiated at a project level. The justifications provided and the context where DSMLs were created are paramount. Future studies should assess the merits and demerits of using a visual modeling language to facilitate PP communications among stakeholders and use evaluation frameworks to identify, modify, or develop them, depending on the scope and goal of the modeling need. UR - https://www.i-jmr.org/2024/1/e55865 UR - http://dx.doi.org/10.2196/55865 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/55865 ER - TY - JOUR AU - Accorsi, Duenhas Tarso Augusto AU - Eduardo, Aires Anderson AU - Baptista, Guilherme Carlos AU - Moreira, Tocci Flavio AU - Morbeck, Albaladejo Renata AU - Köhler, Francine Karen AU - Lima, Amicis Karine de AU - Pedrotti, Sartorato Carlos Henrique PY - 2024/10/25 TI - The Impact of International Classification of Disease?Triggered Prescription Support on Telemedicine: Observational Analysis of Efficiency and Guideline Adherence JO - JMIR Med Inform SP - e56681 VL - 12 KW - telemedicine KW - clinical decision support systems KW - electronic prescriptions KW - guideline adherence KW - consultation efficiency KW - International Classification of Disease?coded prescriptions KW - teleheath KW - eHealth N2 - Background: Integrating decision support systems into telemedicine may optimize consultation efficiency and adherence to clinical guidelines; however, the extent of such effects remains underexplored. Objective: This study aims to evaluate the use of ICD (International Classification of Disease)-coded prescription decision support systems (PDSSs) and the effects of these systems on consultation duration and guideline adherence during telemedicine encounters. Methods: In this retrospective, single-center, observational study conducted from October 2021 to March 2022, adult patients who sought urgent digital care via direct-to-consumer video consultations were included. Physicians had access to current guidelines and could use an ICD-triggered PDSS (which was introduced in January 2022 after a preliminary test in the preceding month) for 26 guideline-based conditions. This study analyzed the impact of implementing automated prescription systems and compared these systems to manual prescription processes in terms of consultation duration and guideline adherence. Results: This study included 10,485 telemedicine encounters involving 9644 patients, with 12,346 prescriptions issued by 290 physicians. Automated prescriptions were used in 5022 (40.67%) of the consultations following system integration. Before introducing decision support, 4497 (36.42%) prescriptions were issued, which increased to 7849 (63.57%) postimplementation. The physician?s average consultation time decreased significantly to 9.5 (SD 5.5) minutes from 11.2 (SD 5.9) minutes after PDSS implementation (P<.001). Of the 12,346 prescriptions, 8683 (70.34%) were aligned with disease-specific international guidelines tailored for telemedicine encounters. Primary medication adherence in accordance with existing guidelines was significantly greater in the decision support group than in the manual group (n=4697, 93.53% vs n=1389, 49.14%; P<.001). Conclusions: Most of the physicians adopted the PDSS, and the results demonstrated the use of the ICD-code system in reducing consultation times and increasing guideline adherence. These systems appear to be valuable for enhancing the efficiency and quality of telemedicine consultations by supporting evidence-based clinical decision-making. UR - https://medinform.jmir.org/2024/1/e56681 UR - http://dx.doi.org/10.2196/56681 UR - http://www.ncbi.nlm.nih.gov/pubmed/39453703 ID - info:doi/10.2196/56681 ER - TY - JOUR AU - Rosenau, Lorenz AU - Gruendner, Julian AU - Kiel, Alexander AU - Köhler, Thomas AU - Schaffer, Bastian AU - Majeed, W. Raphael PY - 2024/10/14 TI - Bridging Data Models in Health Care With a Novel Intermediate Query Format for Feasibility Queries: Mixed Methods Study JO - JMIR Med Inform SP - e58541 VL - 12 KW - feasibility KW - FHIR KW - CQL KW - eligibility criteria KW - clinical research KW - intermediate query format KW - healthcare interoperability KW - cohort definition KW - query KW - queries KW - interoperability KW - interoperable KW - informatics KW - portal KW - portals KW - implementation KW - develop KW - development KW - ontology KW - ontologies KW - JSON N2 - Background: To advance research with clinical data, it is essential to make access to the available data as fast and easy as possible for researchers, which is especially challenging for data from different source systems within and across institutions. Over the years, many research repositories and data standards have been created. One of these is the Fast Healthcare Interoperability Resources (FHIR) standard, used by the German Medical Informatics Initiative (MII) to harmonize and standardize data across university hospitals in Germany. One of the first steps to make these data available is to allow researchers to create feasibility queries to determine the data availability for a specific research question. Given the heterogeneity of different query languages to access different data across and even within standards such as FHIR (eg, CQL and FHIR Search), creating an intermediate query syntax for feasibility queries reduces the complexity of query translation and improves interoperability across different research repositories and query languages. Objective: This study describes the creation and implementation of an intermediate query syntax for feasibility queries and how it integrates into the federated German health research portal (Forschungsdatenportal Gesundheit) and the MII. Methods: We analyzed the requirements for feasibility queries and the feasibility tools that are currently available in research repositories. Based on this analysis, we developed an intermediate query syntax that can be easily translated into different research repository?specific query languages. Results: The resulting Clinical Cohort Definition Language (CCDL) for feasibility queries combines inclusion criteria in a conjunctive normal form and exclusion criteria in a disjunctive normal form, allowing for additional filters like time or numerical restrictions. The inclusion and exclusion results are combined via an expression to specify feasibility queries. We defined a JSON schema for the CCDL, generated an ontology, and demonstrated the use and translatability of the CCDL across multiple studies and real-world use cases. Conclusions: We developed and evaluated a structured query syntax for feasibility queries and demonstrated its use in a real-world example as part of a research platform across 39 German university hospitals. UR - https://medinform.jmir.org/2024/1/e58541 UR - http://dx.doi.org/10.2196/58541 ID - info:doi/10.2196/58541 ER - TY - JOUR AU - Cao, Teng AU - Chen, Zhi AU - Nakayama, Masaharu PY - 2024/10/9 TI - Enhancing the Functionalities of Personal Health Record Systems: Empirical Study Based on the HL7 Personal Health Record System Functional Model Release 1 JO - JMIR Med Inform SP - e56735 VL - 12 KW - fast healthcare interoperability resources KW - logical observation identifiers names and codes KW - personal health record system functional model KW - personal health records N2 - Background: The increasing demand for personal health record (PHR) systems is driven by individuals? desire to actively manage their health care. However, the limited functionality of current PHR systems has affected users? willingness to adopt them, leading to lower-than-expected usage rates. The HL7 (Health Level Seven) PHR System Functional Model (PHR-S FM) was proposed to address this issue, outlining all possible functionalities in PHR systems. Although the PHR-S FM provides a comprehensive theoretical framework, its practical effectiveness and applicability have not been fully explored. Objective: This study aimed to design and develop a tethered PHR prototype in accordance with the guidelines of the PHR-S FM. It sought to explore the feasibility of applying the PHR-S FM in PHR systems by comparing the prototype with the results of previous research. Methods: The PHR-S FM profile was defined to meet broad clinical data management requirements based on previous research. We designed and developed a PHR prototype as a web application using the Fast Healthcare Interoperability Resources R4 (FHIR) and Logical Observation Identifiers Names and Codes (LOINC) coding system for interoperability and data consistency. We validated the prototype using the Synthea dataset, which provided realistic synthetic medical records. In addition, we compared the results produced by the prototype with those of previous studies to evaluate the feasibility and implementation of the PHR-S FM framework. Results: The PHR prototype was developed based on the PHR-S FM profile. We verified its functionality by demonstrating its ability to synchronize data with the FHIR server, effectively managing and displaying various health data types. Validation using the Synthea dataset confirmed the prototype?s accuracy, achieving 100% coverage across 1157 data items. A comparison with the findings of previous studies indicated the feasibility of implementing the PHR-S FM and highlighted areas for future research and improvements. Conclusions: The results of this study offer valuable insights into the potential for practical application and broad adoption of the PHR-S FM in real-world health care settings. UR - https://medinform.jmir.org/2024/1/e56735 UR - http://dx.doi.org/10.2196/56735 ID - info:doi/10.2196/56735 ER - TY - JOUR AU - Chang, Eunsuk AU - Sung, Sumi PY - 2024/10/7 TI - Use of SNOMED CT in Large Language Models: Scoping Review JO - JMIR Med Inform SP - e62924 VL - 12 KW - SNOMED CT KW - ontology KW - knowledge graph KW - large language models KW - natural language processing KW - language models N2 - Background: Large language models (LLMs) have substantially advanced natural language processing (NLP) capabilities but often struggle with knowledge-driven tasks in specialized domains such as biomedicine. Integrating biomedical knowledge sources such as SNOMED CT into LLMs may enhance their performance on biomedical tasks. However, the methodologies and effectiveness of incorporating SNOMED CT into LLMs have not been systematically reviewed. Objective: This scoping review aims to examine how SNOMED CT is integrated into LLMs, focusing on (1) the types and components of LLMs being integrated with SNOMED CT, (2) which contents of SNOMED CT are being integrated, and (3) whether this integration improves LLM performance on NLP tasks. Methods: Following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) guidelines, we searched ACM Digital Library, ACL Anthology, IEEE Xplore, PubMed, and Embase for relevant studies published from 2018 to 2023. Studies were included if they incorporated SNOMED CT into LLM pipelines for natural language understanding or generation tasks. Data on LLM types, SNOMED CT integration methods, end tasks, and performance metrics were extracted and synthesized. Results: The review included 37 studies. Bidirectional Encoder Representations from Transformers and its biomedical variants were the most commonly used LLMs. Three main approaches for integrating SNOMED CT were identified: (1) incorporating SNOMED CT into LLM inputs (28/37, 76%), primarily using concept descriptions to expand training corpora; (2) integrating SNOMED CT into additional fusion modules (5/37, 14%); and (3) using SNOMED CT as an external knowledge retriever during inference (5/37, 14%). The most frequent end task was medical concept normalization (15/37, 41%), followed by entity extraction or typing and classification. While most studies (17/19, 89%) reported performance improvements after SNOMED CT integration, only a small fraction (19/37, 51%) provided direct comparisons. The reported gains varied widely across different metrics and tasks, ranging from 0.87% to 131.66%. However, some studies showed either no improvement or a decline in certain performance metrics. Conclusions: This review demonstrates diverse approaches for integrating SNOMED CT into LLMs, with a focus on using concept descriptions to enhance biomedical language understanding and generation. While the results suggest potential benefits of SNOMED CT integration, the lack of standardized evaluation methods and comprehensive performance reporting hinders definitive conclusions about its effectiveness. Future research should prioritize consistent reporting of performance comparisons and explore more sophisticated methods for incorporating SNOMED CT?s relational structure into LLMs. In addition, the biomedical NLP community should develop standardized evaluation frameworks to better assess the impact of ontology integration on LLM performance. UR - https://medinform.jmir.org/2024/1/e62924 UR - http://dx.doi.org/10.2196/62924 UR - http://www.ncbi.nlm.nih.gov/pubmed/39374057 ID - info:doi/10.2196/62924 ER - TY - JOUR AU - Hu, Zhengyong AU - Wang, Anran AU - Duan, Yifan AU - Zhou, Jiayin AU - Hu, Wanfei AU - Wu, Sizhu PY - 2024/9/30 TI - Toward Better Semantic Interoperability of Data Element Repositories in Medicine: Analysis Study JO - JMIR Med Inform SP - e60293 VL - 12 KW - data element repository KW - FAIR KW - ISO/IEC 11179 KW - metadata KW - semantic interoperability N2 - Background: Data element repositories facilitate high-quality medical data sharing by standardizing data and enhancing semantic interoperability. However, the application of repositories is confined to specific projects and institutions. Objective: This study aims to explore potential issues and promote broader application of data element repositories within the medical field by evaluating and analyzing typical repositories. Methods: Following the inclusion of 5 data element repositories through a literature review, a novel analysis framework consisting of 7 dimensions and 36 secondary indicators was constructed and used for evaluation and analysis. Results: The study?s results delineate the unique characteristics of different repositories and uncover specific issues in their construction. These issues include the absence of data reuse protocols and insufficient information regarding the application scenarios and efficacy of data elements. The repositories fully comply with only 45% (9/20) of the subprinciples for Findable and Reusable in the FAIR principle, while achieving a 90% (19/20 subprinciples) compliance rate for Accessible and 67% (10/15 subprinciples) for Interoperable. Conclusions: The recommendations proposed in this study address the issues to improve the construction and application of repositories, offering valuable insights to data managers, computer experts, and other pertinent stakeholders. UR - https://medinform.jmir.org/2024/1/e60293 UR - http://dx.doi.org/10.2196/60293 UR - http://www.ncbi.nlm.nih.gov/pubmed/39348178 ID - info:doi/10.2196/60293 ER - TY - JOUR AU - Lim, Sachiko AU - Johannesson, Paul PY - 2024/9/26 TI - An Ontology to Bridge the Clinical Management of Patients and Public Health Responses for Strengthening Infectious Disease Surveillance: Design Science Study JO - JMIR Form Res SP - e53711 VL - 8 KW - infectious disease KW - ontology KW - IoT KW - infectious disease surveillance KW - patient monitoring KW - infectious disease management KW - risk analysis KW - early warning KW - data integration KW - semantic interoperability KW - public health N2 - Background: Novel surveillance approaches using digital technologies, including the Internet of Things (IoT), have evolved, enhancing traditional infectious disease surveillance systems by enabling real-time detection of outbreaks and reaching a wider population. However, disparate, heterogenous infectious disease surveillance systems often operate in silos due to a lack of interoperability. As a life-changing clinical use case, the COVID-19 pandemic has manifested that a lack of interoperability can severely inhibit public health responses to emerging infectious diseases. Interoperability is thus critical for building a robust ecosystem of infectious disease surveillance and enhancing preparedness for future outbreaks. The primary enabler for semantic interoperability is ontology. Objective: This study aims to design the IoT-based management of infectious disease ontology (IoT-MIDO) to enhance data sharing and integration of data collected from IoT-driven patient health monitoring, clinical management of individual patients, and disparate heterogeneous infectious disease surveillance. Methods: The ontology modeling approach was chosen for its semantic richness in knowledge representation, flexibility, ease of extensibility, and capability for knowledge inference and reasoning. The IoT-MIDO was developed using the basic formal ontology (BFO) as the top-level ontology. We reused the classes from existing BFO-based ontologies as much as possible to maximize the interoperability with other BFO-based ontologies and databases that rely on them. We formulated the competency questions as requirements for the ontology to achieve the intended goals. Results: We designed an ontology to integrate data from heterogeneous sources, including IoT-driven patient monitoring, clinical management of individual patients, and infectious disease surveillance systems. This integration aims to facilitate the collaboration between clinical care and public health domains. We also demonstrate five use cases using the simplified ontological models to show the potential applications of IoT-MIDO: (1) IoT-driven patient monitoring, risk assessment, early warning, and risk management; (2) clinical management of patients with infectious diseases; (3) epidemic risk analysis for timely response at the public health level; (4) infectious disease surveillance; and (5) transforming patient information into surveillance information. Conclusions: The development of the IoT-MIDO was driven by competency questions. Being able to answer all the formulated competency questions, we successfully demonstrated that our ontology has the potential to facilitate data sharing and integration for orchestrating IoT-driven patient health monitoring in the context of an infectious disease epidemic, clinical patient management, infectious disease surveillance, and epidemic risk analysis. The novelty and uniqueness of the ontology lie in building a bridge to link IoT-based individual patient monitoring and early warning based on patient risk assessment to infectious disease epidemic surveillance at the public health level. The ontology can also serve as a starting point to enable potential decision support systems, providing actionable insights to support public health organizations and practitioners in making informed decisions in a timely manner. UR - https://formative.jmir.org/2024/1/e53711 UR - http://dx.doi.org/10.2196/53711 UR - http://www.ncbi.nlm.nih.gov/pubmed/39325530 ID - info:doi/10.2196/53711 ER - TY - JOUR AU - Ohlsen, Tessa AU - Ingenerf, Josef AU - Essenwanger, Andrea AU - Drenkhahn, Cora PY - 2024/9/17 TI - PCEtoFHIR: Decomposition of Postcoordinated SNOMED CT Expressions for Storage as HL7 FHIR Resources JO - JMIR Med Inform SP - e57853 VL - 12 KW - SNOMED CT KW - HL7 FHIR KW - TermInfo KW - postcoordination KW - semantic interoperability KW - terminology KW - OWL KW - semantic similarity N2 - Background: To ensure interoperability, both structural and semantic standards must be followed. For exchanging medical data between information systems, the structural standard FHIR (Fast Healthcare Interoperability Resources) has recently gained popularity. Regarding semantic interoperability, the reference terminology SNOMED Clinical Terms (SNOMED CT), as a semantic standard, allows for postcoordination, offering advantages over many other vocabularies. These postcoordinated expressions (PCEs) make SNOMED CT an expressive and flexible interlingua, allowing for precise coding of medical facts. However, this comes at the cost of increased complexity, as well as challenges in storage and processing. Additionally, the boundary between semantic (terminology) and structural (information model) standards becomes blurred, leading to what is known as the TermInfo problem. Although often viewed critically, the TermInfo overlap can also be explored for its potential benefits, such as enabling flexible transformation of parts of PCEs. Objective: In this paper, an alternative solution for storing PCEs is presented, which involves combining them with the FHIR data model. Ultimately, all components of a PCE should be expressible solely through precoordinated concepts that are linked to the appropriate elements of the information model. Methods: The approach involves storing PCEs decomposed into their components in alignment with FHIR resources. By utilizing the Web Ontology Language (OWL) to generate an OWL ClassExpression, and combining it with an external reasoner and semantic similarity measures, a precoordinated SNOMED CT concept that most accurately describes the PCE is identified as a Superconcept. In addition, the nonmatching attribute relationships between the Superconcept and the PCE are identified as the ?Delta.? Once SNOMED CT attributes are manually mapped to FHIR elements, FHIRPath expressions can be defined for both the Superconcept and the Delta, allowing the identified precoordinated codes to be stored within FHIR resources. Results: A web application called PCEtoFHIR was developed to implement this approach. In a validation process with 600 randomly selected precoordinated concepts, the formal correctness of the generated OWL ClassExpressions was verified. Additionally, 33 PCEs were used for two separate validation tests. Based on these validations, it was demonstrated that a previously proposed semantic similarity calculation is suitable for determining the Superconcept. Additionally, the 33 PCEs were used to confirm the correct functioning of the entire approach. Furthermore, the FHIR StructureMaps were reviewed and deemed meaningful by FHIR experts. Conclusions: PCEtoFHIR offers services to decompose PCEs for storage within FHIR resources. When creating structure mappings for specific subdomains of SNOMED CT concepts (eg, allergies) to desired FHIR profiles, the use of SNOMED CT Expression Templates has proven highly effective. Domain experts can create templates with appropriate mappings, which can then be easily reused in a constrained manner by end users. UR - https://medinform.jmir.org/2024/1/e57853 UR - http://dx.doi.org/10.2196/57853 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/57853 ER - TY - JOUR AU - Zheng, Chengyi AU - Ackerson, Bradley AU - Qiu, Sijia AU - Sy, S. Lina AU - Daily, Vega Leticia I. AU - Song, Jeannie AU - Qian, Lei AU - Luo, Yi AU - Ku, H. Jennifer AU - Cheng, Yanjun AU - Wu, Jun AU - Tseng, Fu Hung PY - 2024/9/10 TI - Natural Language Processing Versus Diagnosis Code?Based Methods for Postherpetic Neuralgia Identification: Algorithm Development and Validation JO - JMIR Med Inform SP - e57949 VL - 12 KW - postherpetic neuralgia KW - herpes zoster KW - natural language processing KW - electronic health record KW - real-world data KW - artificial intelligence KW - development KW - validation KW - diagnosis KW - EHR KW - algorithm KW - EHR data KW - sensitivity KW - specificity KW - validation data KW - neuralgia KW - recombinant zoster vaccine N2 - Background: Diagnosis codes and prescription data are used in algorithms to identify postherpetic neuralgia (PHN), a debilitating complication of herpes zoster (HZ). Because of the questionable accuracy of codes and prescription data, manual chart review is sometimes used to identify PHN in electronic health records (EHRs), which can be costly and time-consuming. Objective: This study aims to develop and validate a natural language processing (NLP) algorithm for automatically identifying PHN from unstructured EHR data and to compare its performance with that of code-based methods. Methods: This retrospective study used EHR data from Kaiser Permanente Southern California, a large integrated health care system that serves over 4.8 million members. The source population included members aged ?50 years who received an incident HZ diagnosis and accompanying antiviral prescription between 2018 and 2020 and had ?1 encounter within 90?180 days of the incident HZ diagnosis. The study team manually reviewed the EHR and identified PHN cases. For NLP development and validation, 500 and 800 random samples from the source population were selected, respectively. The sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), F-score, and Matthews correlation coefficient (MCC) of NLP and the code-based methods were evaluated using chart-reviewed results as the reference standard. Results: The NLP algorithm identified PHN cases with a 90.9% sensitivity, 98.5% specificity, 82% PPV, and 99.3% NPV. The composite scores of the NLP algorithm were 0.89 (F-score) and 0.85 (MCC). The prevalences of PHN in the validation data were 6.9% (reference standard), 7.6% (NLP), and 5.4%?13.1% (code-based). The code-based methods achieved a 52.7%?61.8% sensitivity, 89.8%?98.4% specificity, 27.6%?72.1% PPV, and 96.3%?97.1% NPV. The F-scores and MCCs ranged between 0.45 and 0.59 and between 0.32 and 0.61, respectively. Conclusions: The automated NLP-based approach identified PHN cases from the EHR with good accuracy. This method could be useful in population-based PHN research. UR - https://medinform.jmir.org/2024/1/e57949 UR - http://dx.doi.org/10.2196/57949 ID - info:doi/10.2196/57949 ER - TY - JOUR AU - Wang, Yueye AU - Han, Xiaotong AU - Li, Cong AU - Luo, Lixia AU - Yin, Qiuxia AU - Zhang, Jian AU - Peng, Guankai AU - Shi, Danli AU - He, Mingguang PY - 2024/8/14 TI - Impact of Gold-Standard Label Errors on Evaluating Performance of Deep Learning Models in Diabetic Retinopathy Screening: Nationwide Real-World Validation Study JO - J Med Internet Res SP - e52506 VL - 26 KW - artificial intelligence KW - diabetic retinopathy KW - diabetes KW - real world KW - deep learning N2 - Background: For medical artificial intelligence (AI) training and validation, human expert labels are considered the gold standard that represents the correct answers or desired outputs for a given data set. These labels serve as a reference or benchmark against which the model?s predictions are compared. Objective: This study aimed to assess the accuracy of a custom deep learning (DL) algorithm on classifying diabetic retinopathy (DR) and further demonstrate how label errors may contribute to this assessment in a nationwide DR-screening program. Methods: Fundus photographs from the Lifeline Express, a nationwide DR-screening program, were analyzed to identify the presence of referable DR using both (1) manual grading by National Health Service England?certificated graders and (2) a DL-based DR-screening algorithm with validated good lab performance. To assess the accuracy of labels, a random sample of images with disagreement between the DL algorithm and the labels was adjudicated by ophthalmologists who were masked to the previous grading results. The error rates of labels in this sample were then used to correct the number of negative and positive cases in the entire data set, serving as postcorrection labels. The DL algorithm?s performance was evaluated against both pre- and postcorrection labels. Results: The analysis included 736,083 images from 237,824 participants. The DL algorithm exhibited a gap between the real-world performance and the lab-reported performance in this nationwide data set, with a sensitivity increase of 12.5% (from 79.6% to 92.5%, P<.001) and a specificity increase of 6.9% (from 91.6% to 98.5%, P<.001). In the random sample, 63.6% (560/880) of negative images and 5.2% (140/2710) of positive images were misclassified in the precorrection human labels. High myopia was the primary reason for misclassifying non-DR images as referable DR images, while laser spots were predominantly responsible for misclassified referable cases. The estimated label error rate for the entire data set was 1.2%. The label correction was estimated to bring about a 12.5% enhancement in the estimated sensitivity of the DL algorithm (P<.001). Conclusions: Label errors based on human image grading, although in a small percentage, can significantly affect the performance evaluation of DL algorithms in real-world DR screening. UR - https://www.jmir.org/2024/1/e52506 UR - http://dx.doi.org/10.2196/52506 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/52506 ER - TY - JOUR AU - Liu, Chang AU - Li, Zhan AU - Li, Jianmin AU - Qu, Yiqian AU - Chang, Ying AU - Han, Qing AU - Cao, Lingyong AU - Lin, Shuyuan PY - 2024/8/2 TI - Research on Traditional Chinese Medicine: Domain Knowledge Graph Completion and Quality Evaluation JO - JMIR Med Inform SP - e55090 VL - 12 KW - graph completion KW - traditional Chinese medicine KW - graph quality evaluation KW - graph representation KW - knowledge graph N2 - Background: Knowledge graphs (KGs) can integrate domain knowledge into a traditional Chinese medicine (TCM) intelligent syndrome differentiation model. However, the quality of current KGs in the TCM domain varies greatly, related to the lack of knowledge graph completion (KGC) and evaluation methods. Objective: This study aims to investigate KGC and evaluation methods tailored for TCM domain knowledge. Methods: In the KGC phase, according to the characteristics of TCM domain knowledge, we proposed a 3-step ?entity-ontology-path? completion approach. This approach uses path reasoning, ontology rule reasoning, and association rules. In the KGC quality evaluation phase, we proposed a 3-dimensional evaluation framework that encompasses completeness, accuracy, and usability, using quantitative metrics such as complex network analysis, ontology reasoning, and graph representation. Furthermore, we compared the impact of different graph representation models on KG usability. Results: In the KGC phase, 52, 107, 27, and 479 triples were added by outlier analysis, rule-based reasoning, association rules, and path-based reasoning, respectively. In addition, rule-based reasoning identified 14 contradictory triples. In the KGC quality evaluation phase, in terms of completeness, KG had higher density and lower sparsity after completion, and there were no contradictory rules within the KG. In terms of accuracy, KG after completion was more consistent with prior knowledge. In terms of usability, the mean reciprocal ranking, mean rank, and hit rate of the first N tail entities predicted by the model (Hits@N) of the TransE, RotatE, DistMult, and ComplEx graph representation models all showed improvement after KGC. Among them, the RotatE model achieved the best representation. Conclusions: The 3-step completion approach can effectively improve the completeness, accuracy, and availability of KGs, and the 3-dimensional evaluation framework can be used for comprehensive KGC evaluation. In the TCM field, the RotatE model performed better at KG representation. UR - https://medinform.jmir.org/2024/1/e55090 UR - http://dx.doi.org/10.2196/55090 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/55090 ER - TY - JOUR AU - Ghasemi, Peyman AU - Lee, Joon PY - 2024/7/26 TI - Unsupervised Feature Selection to Identify Important ICD-10 and ATC Codes for Machine Learning on a Cohort of Patients With Coronary Heart Disease: Retrospective Study JO - JMIR Med Inform SP - e52896 VL - 12 KW - unsupervised feature selection KW - ICD-10 KW - International Classification of Diseases KW - ATC KW - Anatomical Therapeutic Chemical KW - concrete autoencoder KW - Laplacian score KW - unsupervised feature selection for multicluster data KW - autoencoder-inspired unsupervised feature selection KW - principal feature analysis KW - machine learning KW - artificial intelligence KW - case study KW - coronary artery disease KW - artery disease KW - patient cohort KW - artery KW - mortality prediction KW - mortality KW - data set KW - interpretability KW - International Classification of Diseases, Tenth Revision N2 - Background: The application of machine learning in health care often necessitates the use of hierarchical codes such as the International Classification of Diseases (ICD) and Anatomical Therapeutic Chemical (ATC) systems. These codes classify diseases and medications, respectively, thereby forming extensive data dimensions. Unsupervised feature selection tackles the ?curse of dimensionality? and helps to improve the accuracy and performance of supervised learning models by reducing the number of irrelevant or redundant features and avoiding overfitting. Techniques for unsupervised feature selection, such as filter, wrapper, and embedded methods, are implemented to select the most important features with the most intrinsic information. However, they face challenges due to the sheer volume of ICD and ATC codes and the hierarchical structures of these systems. Objective: The objective of this study was to compare several unsupervised feature selection methods for ICD and ATC code databases of patients with coronary artery disease in different aspects of performance and complexity and select the best set of features representing these patients. Methods: We compared several unsupervised feature selection methods for 2 ICD and 1 ATC code databases of 51,506 patients with coronary artery disease in Alberta, Canada. Specifically, we used the Laplacian score, unsupervised feature selection for multicluster data, autoencoder-inspired unsupervised feature selection, principal feature analysis, and concrete autoencoders with and without ICD or ATC tree weight adjustment to select the 100 best features from over 9000 ICD and 2000 ATC codes. We assessed the selected features based on their ability to reconstruct the initial feature space and predict 90-day mortality following discharge. We also compared the complexity of the selected features by mean code level in the ICD or ATC tree and the interpretability of the features in the mortality prediction task using Shapley analysis. Results: In feature space reconstruction and mortality prediction, the concrete autoencoder?based methods outperformed other techniques. Particularly, a weight-adjusted concrete autoencoder variant demonstrated improved reconstruction accuracy and significant predictive performance enhancement, confirmed by DeLong and McNemar tests (P<.05). Concrete autoencoders preferred more general codes, and they consistently reconstructed all features accurately. Additionally, features selected by weight-adjusted concrete autoencoders yielded higher Shapley values in mortality prediction than most alternatives. Conclusions: This study scrutinized 5 feature selection methods in ICD and ATC code data sets in an unsupervised context. Our findings underscore the superiority of the concrete autoencoder method in selecting salient features that represent the entire data set, offering a potential asset for subsequent machine learning research. We also present a novel weight adjustment approach for the concrete autoencoders specifically tailored for ICD and ATC code data sets to enhance the generalizability and interpretability of the selected features. UR - https://medinform.jmir.org/2024/1/e52896 UR - http://dx.doi.org/10.2196/52896 ID - info:doi/10.2196/52896 ER - TY - JOUR AU - Bellmann, Louis AU - Wiederhold, Johannes Alexander AU - Trübe, Leona AU - Twerenbold, Raphael AU - Ückert, Frank AU - Gottfried, Karl PY - 2024/7/24 TI - Introducing Attribute Association Graphs to Facilitate Medical Data Exploration: Development and Evaluation Using Epidemiological Study Data JO - JMIR Med Inform SP - e49865 VL - 12 KW - data exploration KW - cohort studies KW - data visualization KW - big data KW - statistical models KW - medical knowledge KW - data analysis KW - cardiovascular diseases KW - usability N2 - Background: Interpretability and intuitive visualization facilitate medical knowledge generation through big data. In addition, robustness to high-dimensional and missing data is a requirement for statistical approaches in the medical domain. A method tailored to the needs of physicians must meet all the abovementioned criteria. Objective: This study aims to develop an accessible tool for visual data exploration without the need for programming knowledge, adjusting complex parameterizations, or handling missing data. We sought to use statistical analysis using the setting of disease and control cohorts familiar to clinical researchers. We aimed to guide the user by identifying and highlighting data patterns associated with disease and reveal relations between attributes within the data set. Methods: We introduce the attribute association graph, a novel graph structure designed for visual data exploration using robust statistical metrics. The nodes capture frequencies of participant attributes in disease and control cohorts as well as deviations between groups. The edges represent conditional relations between attributes. The graph is visualized using the Neo4j (Neo4j, Inc) data platform and can be interactively explored without the need for technical knowledge. Nodes with high deviations between cohorts and edges of noticeable conditional relationship are highlighted to guide the user during the exploration. The graph is accompanied by a dashboard visualizing variable distributions. For evaluation, we applied the graph and dashboard to the Hamburg City Health Study data set, a large cohort study conducted in the city of Hamburg, Germany. All data structures can be accessed freely by researchers, physicians, and patients. In addition, we developed a user test conducted with physicians incorporating the System Usability Scale, individual questions, and user tasks. Results: We evaluated the attribute association graph and dashboard through an exemplary data analysis of participants with a general cardiovascular disease in the Hamburg City Health Study data set. All results extracted from the graph structure and dashboard are in accordance with findings from the literature, except for unusually low cholesterol levels in participants with cardiovascular disease, which could be induced by medication. In addition, 95% CIs of Pearson correlation coefficients were calculated for all associations identified during the data analysis, confirming the results. In addition, a user test with 10 physicians assessing the usability of the proposed methods was conducted. A System Usability Scale score of 70.5% and average successful task completion of 81.4% were reported. Conclusions: The proposed attribute association graph and dashboard enable intuitive visual data exploration. They are robust to high-dimensional as well as missing data and require no parameterization. The usability for clinicians was confirmed via a user test, and the validity of the statistical results was confirmed by associations known from literature and standard statistical inference. UR - https://medinform.jmir.org/2024/1/e49865 UR - http://dx.doi.org/10.2196/49865 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/49865 ER - TY - JOUR AU - Ji, Hyerim AU - Kim, Seok AU - Sunwoo, Leonard AU - Jang, Sowon AU - Lee, Ho-Young AU - Yoo, Sooyoung PY - 2024/7/12 TI - Integrating Clinical Data and Medical Imaging in Lung Cancer: Feasibility Study Using the Observational Medical Outcomes Partnership Common Data Model Extension JO - JMIR Med Inform SP - e59187 VL - 12 KW - DICOM KW - OMOP KW - CDM KW - lung cancer KW - medical imaging KW - data integration KW - data quality KW - Common Data Model KW - Digital Imaging and Communications in Medicine KW - Observational Medical Outcomes Partnership N2 - Background: Digital transformation, particularly the integration of medical imaging with clinical data, is vital in personalized medicine. The Observational Medical Outcomes Partnership (OMOP) Common Data Model (CDM) standardizes health data. However, integrating medical imaging remains a challenge. Objective: This study proposes a method for combining medical imaging data with the OMOP CDM to improve multimodal research. Methods: Our approach included the analysis and selection of digital imaging and communications in medicine header tags, validation of data formats, and alignment according to the OMOP CDM framework. The Fast Healthcare Interoperability Resources ImagingStudy profile guided our consistency in column naming and definitions. Imaging Common Data Model (I-CDM), constructed using the entity-attribute-value model, facilitates scalable and efficient medical imaging data management. For patients with lung cancer diagnosed between 2010 and 2017, we introduced 4 new tables?IMAGING_STUDY, IMAGING_SERIES, IMAGING_ANNOTATION, and FILEPATH?to standardize various imaging-related data and link to clinical data. Results: This framework underscores the effectiveness of I-CDM in enhancing our understanding of lung cancer diagnostics and treatment strategies. The implementation of the I-CDM tables enabled the structured organization of a comprehensive data set, including 282,098 IMAGING_STUDY, 5,674,425 IMAGING_SERIES, and 48,536 IMAGING_ANNOTATION records, illustrating the extensive scope and depth of the approach. A scenario-based analysis using actual data from patients with lung cancer underscored the feasibility of our approach. A data quality check applying 44 specific rules confirmed the high integrity of the constructed data set, with all checks successfully passed, underscoring the reliability of our findings. Conclusions: These findings indicate that I-CDM can improve the integration and analysis of medical imaging and clinical data. By addressing the challenges in data standardization and management, our approach contributes toward enhancing diagnostics and treatment strategies. Future research should expand the application of I-CDM to diverse disease populations and explore its wide-ranging utility for medical conditions. UR - https://medinform.jmir.org/2024/1/e59187 UR - http://dx.doi.org/10.2196/59187 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/59187 ER - TY - JOUR AU - Herman Bernardim Andrade, Gabriel AU - Yada, Shuntaro AU - Aramaki, Eiji PY - 2024/7/2 TI - Is Boundary Annotation Necessary? Evaluating Boundary-Free Approaches to Improve Clinical Named Entity Annotation Efficiency: Case Study JO - JMIR Med Inform SP - e59680 VL - 12 KW - natural language processing KW - named entity recognition KW - information extraction KW - text annotation KW - entity boundaries KW - lenient annotation KW - case reports KW - annotation KW - case study KW - medical case report KW - efficiency KW - model KW - model performance KW - dataset KW - Japan KW - Japanese KW - entity KW - clinical domain KW - clinical N2 - Background: Named entity recognition (NER) is a fundamental task in natural language processing. However, it is typically preceded by named entity annotation, which poses several challenges, especially in the clinical domain. For instance, determining entity boundaries is one of the most common sources of disagreements between annotators due to questions such as whether modifiers or peripheral words should be annotated. If unresolved, these can induce inconsistency in the produced corpora, yet, on the other hand, strict guidelines or adjudication sessions can further prolong an already slow and convoluted process. Objective: The aim of this study is to address these challenges by evaluating 2 novel annotation methodologies, lenient span and point annotation, aiming to mitigate the difficulty of precisely determining entity boundaries. Methods: We evaluate their effects through an annotation case study on a Japanese medical case report data set. We compare annotation time, annotator agreement, and the quality of the produced labeling and assess the impact on the performance of an NER system trained on the annotated corpus. Results: We saw significant improvements in the labeling process efficiency, with up to a 25% reduction in overall annotation time and even a 10% improvement in annotator agreement compared to the traditional boundary-strict approach. However, even the best-achieved NER model presented some drop in performance compared to the traditional annotation methodology. Conclusions: Our findings demonstrate a balance between annotation speed and model performance. Although disregarding boundary information affects model performance to some extent, this is counterbalanced by significant reductions in the annotator?s workload and notable improvements in the speed of the annotation process. These benefits may prove valuable in various applications, offering an attractive compromise for developers and researchers. UR - https://medinform.jmir.org/2024/1/e59680 UR - http://dx.doi.org/10.2196/59680 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/59680 ER - TY - JOUR AU - Spoladore, Daniele AU - Colombo, Vera AU - Fumagalli, Alessia AU - Tosi, Martina AU - Lorenzini, Cecilia Erna AU - Sacco, Marco PY - 2024/6/26 TI - An Ontology-Based Decision Support System for Tailored Clinical Nutrition Recommendations for Patients With Chronic Obstructive Pulmonary Disease: Development and Acceptability Study JO - JMIR Med Inform SP - e50980 VL - 12 KW - ontology-based decision support system KW - nutritional recommendation KW - chronic obstructive pulmonary disease KW - clinical decision support system KW - pulmonary rehabilitation N2 - Background: Chronic obstructive pulmonary disease (COPD) is a chronic condition among the main causes of morbidity and mortality worldwide, representing a burden on health care systems. Scientific literature highlights that nutrition is pivotal in respiratory inflammatory processes connected to COPD, including exacerbations. Patients with COPD have an increased risk of developing nutrition-related comorbidities, such as diabetes, cardiovascular diseases, and malnutrition. Moreover, these patients often manifest sarcopenia and cachexia. Therefore, an adequate nutritional assessment and therapy are essential to help individuals with COPD in managing the progress of the disease. However, the role of nutrition in pulmonary rehabilitation (PR) programs is often underestimated due to a lack of resources and dedicated services, mostly because pneumologists may lack the specialized training for such a discipline. Objective: This work proposes a novel knowledge-based decision support system to support pneumologists in considering nutritional aspects in PR. The system provides clinicians with patient-tailored dietary recommendations leveraging expert knowledge. Methods: The expert knowledge?acquired from experts and clinical literature?was formalized in domain ontologies and rules, which were developed leveraging the support of Italian clinicians with expertise in the rehabilitation of patients with COPD. Thus, by following an agile ontology engineering methodology, the relevant formal ontologies were developed to act as a backbone for an application targeted at pneumologists. The recommendations provided by the decision support system were validated by a group of nutrition experts, whereas the acceptability of such an application in the context of PR was evaluated by pneumologists. Results: A total of 7 dieticians (mean age 46.60, SD 13.35 years) were interviewed to assess their level of agreement with the decision support system?s recommendations by evaluating 5 patients? health conditions. The preliminary results indicate that the system performed more than adequately (with an overall average score of 4.23, SD 0.52 out of 5 points), providing meaningful and safe recommendations in compliance with clinical practice. With regard to the acceptability of the system by lung specialists (mean age 44.71, SD 11.94 years), the usefulness and relevance of the proposed solution were extremely positive?the scores on each of the perceived usefulness subscales of the technology acceptance model 3 were 4.86 (SD 0.38) out of 5 points, whereas the score on the intention to use subscale was 4.14 (SD 0.38) out of 5 points. Conclusions: Although designed for the Italian clinical context, the proposed system can be adapted for any other national clinical context by modifying the domain ontologies, thus providing a multidisciplinary approach to the management of patients with COPD. UR - https://medinform.jmir.org/2024/1/e50980 UR - http://dx.doi.org/10.2196/50980 UR - http://www.ncbi.nlm.nih.gov/pubmed/38922666 ID - info:doi/10.2196/50980 ER - TY - JOUR AU - Luo, Xufei AU - Chen, Fengxian AU - Zhu, Di AU - Wang, Ling AU - Wang, Zijun AU - Liu, Hui AU - Lyu, Meng AU - Wang, Ye AU - Wang, Qi AU - Chen, Yaolong PY - 2024/6/25 TI - Potential Roles of Large Language Models in the Production of Systematic Reviews and Meta-Analyses JO - J Med Internet Res SP - e56780 VL - 26 KW - large language model KW - ChatGPT KW - systematic review KW - chatbot KW - meta-analysis UR - https://www.jmir.org/2024/1/e56780 UR - http://dx.doi.org/10.2196/56780 UR - http://www.ncbi.nlm.nih.gov/pubmed/38819655 ID - info:doi/10.2196/56780 ER - TY - JOUR AU - Lin, Z. Rebecca AU - Amith, Tuan Muhammad AU - Wang, X. Cynthia AU - Strickley, John AU - Tao, Cui PY - 2024/6/21 TI - Dermoscopy Differential Diagnosis Explorer (D3X) Ontology to Aggregate and Link Dermoscopic Patterns to Differential Diagnoses: Development and Usability Study JO - JMIR Med Inform SP - e49613 VL - 12 KW - medical informatics KW - biomedical ontology KW - ontology KW - ontologies KW - vocabulary KW - OWL KW - web ontology language KW - skin KW - semiotic KW - web app KW - web application KW - visual KW - visualization KW - dermoscopic KW - diagnosis KW - diagnoses KW - diagnostic KW - information storage KW - information retrieval KW - skin lesion KW - skin diseases KW - dermoscopy differential diagnosis explorer KW - dermatology KW - dermoscopy KW - differential diagnosis KW - information storage and retrieval N2 - Background: Dermoscopy is a growing field that uses microscopy to allow dermatologists and primary care physicians to identify skin lesions. For a given skin lesion, a wide variety of differential diagnoses exist, which may be challenging for inexperienced users to name and understand. Objective: In this study, we describe the creation of the dermoscopy differential diagnosis explorer (D3X), an ontology linking dermoscopic patterns to differential diagnoses. Methods: Existing ontologies that were incorporated into D3X include the elements of visuals ontology and dermoscopy elements of visuals ontology, which connect visual features to dermoscopic patterns. A list of differential diagnoses for each pattern was generated from the literature and in consultation with domain experts. Open-source images were incorporated from DermNet, Dermoscopedia, and open-access research papers. Results: D3X was encoded in the OWL 2 web ontology language and includes 3041 logical axioms, 1519 classes, 103 object properties, and 20 data properties. We compared D3X with publicly available ontologies in the dermatology domain using a semiotic theory?driven metric to measure the innate qualities of D3X with others. The results indicate that D3X is adequately comparable with other ontologies of the dermatology domain. Conclusions: The D3X ontology is a resource that can link and integrate dermoscopic differential diagnoses and supplementary information with existing ontology-based resources. Future directions include developing a web application based on D3X for dermoscopy education and clinical practice. UR - https://medinform.jmir.org/2024/1/e49613 UR - http://dx.doi.org/10.2196/49613 UR - http://www.ncbi.nlm.nih.gov/pubmed/38904996 ID - info:doi/10.2196/49613 ER - TY - JOUR AU - Nyein, Phyu Kyi AU - Condron, Claire PY - 2024/6/17 TI - Communication and Contextual Factors in Robotic-Assisted Surgical Teams: Protocol for Developing a Taxonomy JO - JMIR Res Protoc SP - e54910 VL - 13 KW - communication KW - teams KW - robotic surgery KW - robotic-assisted KW - simulation N2 - Background: Robotic-assisted surgery (RAS) has been rapidly integrated into surgical practice in the past few decades. The setup in the operating theater for RAS differs from that for open or laparoscopic surgery such that the operating surgeon sits at a console separate from the rest of the surgical team and the patient. Communication and team dynamics are altered due to this physical separation and visual barriers imposed by the robotic equipment. However, the factors that might comprise effective communication among members of RAS teams and the contextual factors that facilitate or inhibit effective communication in RAS remain unknown. Objective: We aim to develop a taxonomy of communication behaviors and contextual factors that influence communication in RAS teams. We also aim to examine the patterns of communication behaviors based on gender. Methods: We will first perform a scoping review on communication in RAS to develop a preliminary taxonomy of communication based on the existing literature. We will then conduct semistructured interviews with RAS team members, including the surgeon, assisting surgeon or trainee, bedside or first assistant, nurses, and anesthetists. Participants will represent different disciplines, including urology, general surgery, and gynecology, and have a range of experiences in RAS. We will use a reflexive thematic analysis to analyze the data and further refine the taxonomy. We will also observe live robotic surgeries at Royal College of Surgeons in Ireland (RCSI)?affiliated hospitals. We will observe varying lengths and conditions of RAS procedures to a capture a wide range of communication behaviors and contextual factors to help finalize the taxonomy. Although we anticipate conducting 30 interviews and 30 observations, we will collect data until we achieve data sufficiency. We will conduct data collection in parallel with data analysis such that if we identify a new behavior in an interview, we will follow up with questions related to that behavior in additional interviews and/or observations. Results: The taxonomy from this project will include a list of actionable communication behaviors, contextual factors, their descriptions, and examples. As of May 2024, this project has been approved by the RCSI Research and Ethics Committee. Data collection started in June 2024 and will continue throughout the year. We plan to publish the findings as meaningful results emerge in our data analysis in 2024 and 2025. Conclusions: The results from this project will be used to observe and train surgical teams in a simulated environment to effectively communicate with each other and prevent communication breakdowns. The developed taxonomy will also add to the knowledge base on the role of gender in communication in RAS and produce recommendations that can be incorporated into training. Overall, this project will contribute to the improvement of communication skills of surgical teams and the quality and safety of patient care. International Registered Report Identifier (IRRID): PRR1-10.2196/54910 UR - https://www.researchprotocols.org/2024/1/e54910 UR - http://dx.doi.org/10.2196/54910 UR - http://www.ncbi.nlm.nih.gov/pubmed/38885018 ID - info:doi/10.2196/54910 ER - TY - JOUR AU - Stellmach, Caroline AU - Hopff, Marie Sina AU - Jaenisch, Thomas AU - Nunes de Miranda, Marina Susana AU - Rinaldi, Eugenia AU - PY - 2024/6/10 TI - Creation of Standardized Common Data Elements for Diagnostic Tests in Infectious Disease Studies: Semantic and Syntactic Mapping JO - J Med Internet Res SP - e50049 VL - 26 KW - core data element KW - CDE KW - case report form KW - CRF KW - interoperability KW - semantic standards KW - infectious disease KW - diagnostic test KW - covid19 KW - COVID-19 KW - mpox KW - ZIKV KW - patient data KW - data model KW - syntactic interoperability KW - clinical data KW - FHIR KW - SNOMED CT KW - LOINC KW - virus infection KW - common element N2 - Background: It is necessary to harmonize and standardize data variables used in case report forms (CRFs) of clinical studies to facilitate the merging and sharing of the collected patient data across several clinical studies. This is particularly true for clinical studies that focus on infectious diseases. Public health may be highly dependent on the findings of such studies. Hence, there is an elevated urgency to generate meaningful, reliable insights, ideally based on a high sample number and quality data. The implementation of core data elements and the incorporation of interoperability standards can facilitate the creation of harmonized clinical data sets. Objective: This study?s objective was to compare, harmonize, and standardize variables focused on diagnostic tests used as part of CRFs in 6 international clinical studies of infectious diseases in order to, ultimately, then make available the panstudy common data elements (CDEs) for ongoing and future studies to foster interoperability and comparability of collected data across trials. Methods: We reviewed and compared the metadata that comprised the CRFs used for data collection in and across all 6 infectious disease studies under consideration in order to identify CDEs. We examined the availability of international semantic standard codes within the Systemized Nomenclature of Medicine - Clinical Terms, the National Cancer Institute Thesaurus, and the Logical Observation Identifiers Names and Codes system for the unambiguous representation of diagnostic testing information that makes up the CDEs. We then proposed 2 data models that incorporate semantic and syntactic standards for the identified CDEs. Results: Of 216 variables that were considered in the scope of the analysis, we identified 11 CDEs to describe diagnostic tests (in particular, serology and sequencing) for infectious diseases: viral lineage/clade; test date, type, performer, and manufacturer; target gene; quantitative and qualitative results; and specimen identifier, type, and collection date. Conclusions: The identification of CDEs for infectious diseases is the first step in facilitating the exchange and possible merging of a subset of data across clinical studies (and with that, large research projects) for possible shared analysis to increase the power of findings. The path to harmonization and standardization of clinical study data in the interest of interoperability can be paved in 2 ways. First, a map to standard terminologies ensures that each data element?s (variable?s) definition is unambiguous and that it has a single, unique interpretation across studies. Second, the exchange of these data is assisted by ?wrapping? them in a standard exchange format, such as Fast Health care Interoperability Resources or the Clinical Data Interchange Standards Consortium?s Clinical Data Acquisition Standards Harmonization Model. UR - https://www.jmir.org/2024/1/e50049 UR - http://dx.doi.org/10.2196/50049 UR - http://www.ncbi.nlm.nih.gov/pubmed/38857066 ID - info:doi/10.2196/50049 ER - TY - JOUR AU - Ohno, Yukiko AU - Kato, Riri AU - Ishikawa, Haruki AU - Nishiyama, Tomohiro AU - Isawa, Minae AU - Mochizuki, Mayumi AU - Aramaki, Eiji AU - Aomori, Tohru PY - 2024/6/4 TI - Using the Natural Language Processing System Medical Named Entity Recognition-Japanese to Analyze Pharmaceutical Care Records: Natural Language Processing Analysis JO - JMIR Form Res SP - e55798 VL - 8 KW - natural language processing KW - NLP KW - named entity recognition KW - pharmaceutical care records KW - machine learning KW - cefazolin sodium KW - electronic medical record KW - EMR KW - extraction KW - Japanese N2 - Background: Large language models have propelled recent advances in artificial intelligence technology, facilitating the extraction of medical information from unstructured data such as medical records. Although named entity recognition (NER) is used to extract data from physicians? records, it has yet to be widely applied to pharmaceutical care records. Objective: In this study, we aimed to investigate the feasibility of automatic extraction of the information regarding patients? diseases and symptoms from pharmaceutical care records. The verification was performed using Medical Named Entity Recognition-Japanese (MedNER-J), a Japanese disease-extraction system designed for physicians? records. Methods: MedNER-J was applied to subjective, objective, assessment, and plan data from the care records of 49 patients who received cefazolin sodium injection at Keio University Hospital between April 2018 and March 2019. The performance of MedNER-J was evaluated in terms of precision, recall, and F1-score. Results: The F1-scores of NER for subjective, objective, assessment, and plan data were 0.46, 0.70, 0.76, and 0.35, respectively. In NER and positive-negative classification, the F1-scores were 0.28, 0.39, 0.64, and 0.077, respectively. The F1-scores of NER for objective (0.70) and assessment data (0.76) were higher than those for subjective and plan data, which supported the superiority of NER performance for objective and assessment data. This might be because objective and assessment data contained many technical terms, similar to the training data for MedNER-J. Meanwhile, the F1-score of NER and positive-negative classification was high for assessment data alone (F1-score=0.64), which was attributed to the similarity of its description format and contents to those of the training data. Conclusions: MedNER-J successfully read pharmaceutical care records and showed the best performance for assessment data. However, challenges remain in analyzing records other than assessment data. Therefore, it will be necessary to reinforce the training data for subjective data in order to apply the system to pharmaceutical care records. UR - https://formative.jmir.org/2024/1/e55798 UR - http://dx.doi.org/10.2196/55798 UR - http://www.ncbi.nlm.nih.gov/pubmed/38833694 ID - info:doi/10.2196/55798 ER - TY - JOUR AU - Janssen, Anna AU - Donnelly, Candice AU - Shaw, Tim PY - 2024/5/31 TI - A Taxonomy for Health Information Systems JO - J Med Internet Res SP - e47682 VL - 26 KW - eHealth KW - digital health KW - electronic health data KW - data revolution KW - actionable data KW - mobile phone UR - https://www.jmir.org/2024/1/e47682 UR - http://dx.doi.org/10.2196/47682 UR - http://www.ncbi.nlm.nih.gov/pubmed/38820575 ID - info:doi/10.2196/47682 ER - TY - JOUR AU - Lee, Vien V. AU - van der Lubbe, C. Stephanie C. AU - Goh, Hoon Lay AU - Valderas, Maria Jose PY - 2024/5/31 TI - Harnessing ChatGPT for Thematic Analysis: Are We Ready? JO - J Med Internet Res SP - e54974 VL - 26 KW - ChatGPT KW - thematic analysis KW - natural language processing KW - NLP KW - medical research KW - qualitative research KW - qualitative data KW - technology KW - viewpoint KW - efficiency UR - https://www.jmir.org/2024/1/e54974 UR - http://dx.doi.org/10.2196/54974 UR - http://www.ncbi.nlm.nih.gov/pubmed/38819896 ID - info:doi/10.2196/54974 ER - TY - JOUR AU - Yoon, Dukyong AU - Han, Changho AU - Kim, Won Dong AU - Kim, Songsoo AU - Bae, SungA AU - Ryu, An Jee AU - Choi, Yujin PY - 2024/5/31 TI - Redefining Health Care Data Interoperability: Empirical Exploration of Large Language Models in Information Exchange JO - J Med Internet Res SP - e56614 VL - 26 KW - health care interoperability KW - large language models KW - medical data transformation KW - data standardization KW - text-based N2 - Background: Efficient data exchange and health care interoperability are impeded by medical records often being in nonstandardized or unstructured natural language format. Advanced language models, such as large language models (LLMs), may help overcome current challenges in information exchange. Objective: This study aims to evaluate the capability of LLMs in transforming and transferring health care data to support interoperability. Methods: Using data from the Medical Information Mart for Intensive Care III and UK Biobank, the study conducted 3 experiments. Experiment 1 assessed the accuracy of transforming structured laboratory results into unstructured format. Experiment 2 explored the conversion of diagnostic codes between the coding frameworks of the ICD-9-CM (International Classification of Diseases, Ninth Revision, Clinical Modification), and Systematized Nomenclature of Medicine Clinical Terms (SNOMED-CT) using a traditional mapping table and a text-based approach facilitated by the LLM ChatGPT. Experiment 3 focused on extracting targeted information from unstructured records that included comprehensive clinical information (discharge notes). Results: The text-based approach showed a high conversion accuracy in transforming laboratory results (experiment 1) and an enhanced consistency in diagnostic code conversion, particularly for frequently used diagnostic names, compared with the traditional mapping approach (experiment 2). In experiment 3, the LLM showed a positive predictive value of 87.2% in extracting generic drug names. Conclusions: This study highlighted the potential role of LLMs in significantly improving health care data interoperability, demonstrated by their high accuracy and efficiency in data transformation and exchange. The LLMs hold vast potential for enhancing medical data exchange without complex standardization for medical terms and data structure. UR - https://www.jmir.org/2024/1/e56614 UR - http://dx.doi.org/10.2196/56614 UR - http://www.ncbi.nlm.nih.gov/pubmed/38819879 ID - info:doi/10.2196/56614 ER - TY - JOUR AU - Invernici, Francesco AU - Bernasconi, Anna AU - Ceri, Stefano PY - 2024/5/30 TI - Searching COVID-19 Clinical Research Using Graph Queries: Algorithm Development and Validation JO - J Med Internet Res SP - e52655 VL - 26 KW - big data corpus KW - clinical research KW - co-occurrence network KW - COVID-19 Open Research Dataset KW - CORD-19 KW - graph search KW - Named Entity Recognition KW - Neo4j KW - text mining N2 - Background: Since the beginning of the COVID-19 pandemic, >1 million studies have been collected within the COVID-19 Open Research Dataset, a corpus of manuscripts created to accelerate research against the disease. Their related abstracts hold a wealth of information that remains largely unexplored and difficult to search due to its unstructured nature. Keyword-based search is the standard approach, which allows users to retrieve the documents of a corpus that contain (all or some of) the words in a target list. This type of search, however, does not provide visual support to the task and is not suited to expressing complex queries or compensating for missing specifications. Objective: This study aims to consider small graphs of concepts and exploit them for expressing graph searches over existing COVID-19?related literature, leveraging the increasing use of graphs to represent and query scientific knowledge and providing a user-friendly search and exploration experience. Methods: We considered the COVID-19 Open Research Dataset corpus and summarized its content by annotating the publications? abstracts using terms selected from the Unified Medical Language System and the Ontology of Coronavirus Infectious Disease. Then, we built a co-occurrence network that includes all relevant concepts mentioned in the corpus, establishing connections when their mutual information is relevant. A sophisticated graph query engine was built to allow the identification of the best matches of graph queries on the network. It also supports partial matches and suggests potential query completions using shortest paths. Results: We built a large co-occurrence network, consisting of 128,249 entities and 47,198,965 relationships; the GRAPH-SEARCH interface allows users to explore the network by formulating or adapting graph queries; it produces a bibliography of publications, which are globally ranked; and each publication is further associated with the specific parts of the query that it explains, thereby allowing the user to understand each aspect of the matching. Conclusions: Our approach supports the process of query formulation and evidence search upon a large text corpus; it can be reapplied to any scientific domain where documents corpora and curated ontologies are made available. UR - https://www.jmir.org/2024/1/e52655 UR - http://dx.doi.org/10.2196/52655 UR - http://www.ncbi.nlm.nih.gov/pubmed/38814687 ID - info:doi/10.2196/52655 ER - TY - JOUR AU - Chiu, Keith Wan Hang AU - Ko, Koel Wei Sum AU - Cho, Shing William Chi AU - Hui, Joanne Sin Yu AU - Chan, Lawrence Wing Chi AU - Kuo, D. Michael PY - 2024/5/13 TI - Evaluating the Diagnostic Performance of Large Language Models on Complex Multimodal Medical Cases JO - J Med Internet Res SP - e53724 VL - 26 KW - large language model KW - hospital KW - health center KW - Massachusetts KW - statistical analysis KW - chi-square KW - ANOVA KW - clinician KW - physician KW - performance KW - proficiency KW - disease etiology UR - https://www.jmir.org/2024/1/e53724 UR - http://dx.doi.org/10.2196/53724 UR - http://www.ncbi.nlm.nih.gov/pubmed/38739441 ID - info:doi/10.2196/53724 ER - TY - JOUR AU - Senior, Rashaud AU - Tsai, Timothy AU - Ratliff, William AU - Nadler, Lisa AU - Balu, Suresh AU - Malcolm, Elizabeth AU - McPeek Hinz, Eugenia PY - 2024/5/9 TI - Evaluation of SNOMED CT Grouper Accuracy and Coverage in Organizing the Electronic Health Record Problem List by Clinical System: Observational Study JO - JMIR Med Inform SP - e51274 VL - 12 KW - electronic health record KW - problem List KW - problem list organization KW - problem list management KW - SNOMED CT KW - SNOMED CT Groupers KW - Systematized Nomenclature of Medicine KW - clinical term KW - ICD-10 KW - International Classification of Diseases N2 - Background: The problem list (PL) is a repository of diagnoses for patients? medical conditions and health-related issues. Unfortunately, over time, our PLs have become overloaded with duplications, conflicting entries, and no-longer-valid diagnoses. The lack of a standardized structure for review adds to the challenges of clinical use. Previously, our default electronic health record (EHR) organized the PL primarily via alphabetization, with other options available, for example, organization by clinical systems or priority settings. The system?s PL was built with limited groupers, resulting in many diagnoses that were inconsistent with the expected clinical systems or not associated with any clinical systems at all. As a consequence of these limited EHR configuration options, our PL organization has poorly supported clinical use over time, particularly as the number of diagnoses on the PL has increased. Objective: We aimed to measure the accuracy of sorting PL diagnoses into PL system groupers based on Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) concept groupers implemented in our EHR. Methods: We transformed and developed 21 system- or condition-based groupers, using 1211 SNOMED CT hierarchal concepts refined with Boolean logic, to reorganize the PL in our EHR. To evaluate the clinical utility of our new groupers, we extracted all diagnoses on the PLs from a convenience sample of 50 patients with 3 or more encounters in the previous year. To provide a spectrum of clinical diagnoses, we included patients from all ages and divided them by sex in a deidentified format. Two physicians independently determined whether each diagnosis was correctly attributed to the expected clinical system grouper. Discrepancies were discussed, and if no consensus was reached, they were adjudicated by a third physician. Descriptive statistics and Cohen ? statistics for interrater reliability were calculated. Results: Our 50-patient sample had a total of 869 diagnoses (range 4-59; median 12, IQR 9-24). The reviewers initially agreed on 821 system attributions. Of the remaining 48 items, 16 required adjudication with the tie-breaking third physician. The calculated ? statistic was 0.7. The PL groupers appropriately associated diagnoses to the expected clinical system with a sensitivity of 97.6%, a specificity of 58.7%, a positive predictive value of 96.8%, and an F1-score of 0.972. Conclusions: We found that PL organization by clinical specialty or condition using SNOMED CT concept groupers accurately reflects clinical systems. Our system groupers were subsequently adopted by our vendor EHR in their foundation system for PL organization. UR - https://medinform.jmir.org/2024/1/e51274 UR - http://dx.doi.org/10.2196/51274 ID - info:doi/10.2196/51274 ER - TY - JOUR AU - Maripuri, Monika AU - Dey, Andrew AU - Honerlaw, Jacqueline AU - Hong, Chuan AU - Ho, Yuk-Lam AU - Tanukonda, Vidisha AU - Chen, W. Alicia AU - Panickan, Ayakulangara Vidul AU - Wang, Xuan AU - Zhang, G. Harrison AU - Yang, Doris AU - Samayamuthu, Jebathilagam Malarkodi AU - Morris, Michele AU - Visweswaran, Shyam AU - Beaulieu-Jones, Brendin AU - Ramoni, Rachel AU - Muralidhar, Sumitra AU - Gaziano, Michael J. AU - Liao, Katherine AU - Xia, Zongqi AU - Brat, A. Gabriel AU - Cai, Tianxi AU - Cho, Kelly PY - 2024/5/3 TI - Characterization of Post?COVID-19 Definitions and Clinical Coding Practices: Longitudinal Study JO - Online J Public Health Inform SP - e53445 VL - 16 KW - veterans KW - long COVID-19 KW - postacute sequelae of SARS-CoV-2 KW - PASC KW - International Classification of Diseases KW - U09.9 ICD-10 code KW - algorithm validation KW - chart review KW - electronic health records KW - COVID-19 N2 - Background: Post?COVID-19 condition (colloquially known as ?long COVID-19?) characterized as postacute sequelae of SARS-CoV-2 has no universal clinical case definition. Recent efforts have focused on understanding long COVID-19 symptoms, and electronic health record (EHR) data provide a unique resource for understanding this condition. The introduction of the International Classification of Diseases, Tenth Revision (ICD-10) code U09.9 for ?Post COVID-19 condition, unspecified? to identify patients with long COVID-19 has provided a method of evaluating this condition in EHRs; however, the accuracy of this code is unclear. Objective: This study aimed to characterize the utility and accuracy of the U09.9 code across 3 health care systems?the Veterans Health Administration, the Beth Israel Deaconess Medical Center, and the University of Pittsburgh Medical Center?against patients identified with long COVID-19 via a chart review by operationalizing the World Health Organization (WHO) and Centers for Disease Control and Prevention (CDC) definitions. Methods: Patients who were COVID-19 positive with either a U07.1 ICD-10 code or positive polymerase chain reaction test within these health care systems were identified for chart review. Among this cohort, we sampled patients based on two approaches: (1) with a U09.9 code and (2) without a U09.9 code but with a new onset long COVID-19?related ICD-10 code, which allows us to assess the sensitivity of the U09.9 code. To operationalize the long COVID-19 definition based on health agency guidelines, symptoms were grouped into a ?core? cluster of 11 commonly reported symptoms among patients with long COVID-19 and an extended cluster that captured all other symptoms by disease domain. Patients having ?2 symptoms persisting for ?60 days that were new onset after their COVID-19 infection, with ?1 symptom in the core cluster, were labeled as having long COVID-19 per chart review. The code?s performance was compared across 3 health care systems and across different time periods of the pandemic. Results: Overall, 900 patient charts were reviewed across 3 health care systems. The prevalence of long COVID-19 among the cohort with the U09.9 ICD-10 code based on the operationalized WHO definition was between 23.2% and 62.4% across these health care systems. We also evaluated a less stringent version of the WHO definition and the CDC definition and observed an increase in the prevalence of long COVID-19 at all 3 health care systems. Conclusions: This is one of the first studies to evaluate the U09.9 code against a clinical case definition for long COVID-19, as well as the first to apply this definition to EHR data using a chart review approach on a nationwide cohort across multiple health care systems. This chart review approach can be implemented at other EHR systems to further evaluate the utility and performance of the U09.9 code. UR - https://ojphi.jmir.org/2024/1/e53445 UR - http://dx.doi.org/10.2196/53445 UR - http://www.ncbi.nlm.nih.gov/pubmed/38700929 ID - info:doi/10.2196/53445 ER - TY - JOUR AU - Palojoki, Sari AU - Lehtonen, Lasse AU - Vuokko, Riikka PY - 2024/4/25 TI - Semantic Interoperability of Electronic Health Records: Systematic Review of Alternative Approaches for Enhancing Patient Information Availability JO - JMIR Med Inform SP - e53535 VL - 12 KW - electronic health record KW - health records KW - EHR KW - EHRs KW - semantic KW - health care data KW - semantic interoperability KW - interoperability KW - standardize KW - standardized KW - standardization KW - cross-border data exchange KW - systematic review KW - synthesis KW - syntheses KW - review methods KW - review methodology KW - search KW - searches KW - searching KW - systematic KW - data exchange KW - information sharing KW - ontology KW - ontologies KW - terminology KW - terminologies KW - standard KW - standards KW - classification KW - PRISMA KW - data sharing KW - Preferred Reporting Items for Systematic Reviews and Meta-Analyses N2 - Background: Semantic interoperability facilitates the exchange of and access to health data that are being documented in electronic health records (EHRs) with various semantic features. The main goals of semantic interoperability development entail patient data availability and use in diverse EHRs without a loss of meaning. Internationally, current initiatives aim to enhance semantic development of EHR data and, consequently, the availability of patient data. Interoperability between health information systems is among the core goals of the European Health Data Space regulation proposal and the World Health Organization?s Global Strategy on Digital Health 2020-2025. Objective: To achieve integrated health data ecosystems, stakeholders need to overcome challenges of implementing semantic interoperability elements. To research the available scientific evidence on semantic interoperability development, we defined the following research questions: What are the key elements of and approaches for building semantic interoperability integrated in EHRs? What kinds of goals are driving the development? and What kinds of clinical benefits are perceived following this development? Methods: Our research questions focused on key aspects and approaches for semantic interoperability and on possible clinical and semantic benefits of these choices in the context of EHRs. Therefore, we performed a systematic literature review in PubMed by defining our study framework based on previous research. Results: Our analysis consisted of 14 studies where data models, ontologies, terminologies, classifications, and standards were applied for building interoperability. All articles reported clinical benefits of the selected approach to enhancing semantic interoperability. We identified 3 main categories: increasing the availability of data for clinicians (n=6, 43%), increasing the quality of care (n=4, 29%), and enhancing clinical data use and reuse for varied purposes (n=4, 29%). Regarding semantic development goals, data harmonization and developing semantic interoperability between different EHRs was the largest category (n=8, 57%). Enhancing health data quality through standardization (n=5, 36%) and developing EHR-integrated tools based on interoperable data (n=1, 7%) were the other identified categories. The results were closely coupled with the need to build usable and computable data out of heterogeneous medical information that is accessible through various EHRs and databases (eg, registers). Conclusions: When heading toward semantic harmonization of clinical data, more experiences and analyses are needed to assess how applicable the chosen solutions are for semantic interoperability of health care data. Instead of promoting a single approach, semantic interoperability should be assessed through several levels of semantic requirements A dual model or multimodel approach is possibly usable to address different semantic interoperability issues during development. The objectives of semantic interoperability are to be achieved in diffuse and disconnected clinical care environments. Therefore, approaches for enhancing clinical data availability should be well prepared, thought out, and justified to meet economically sustainable and long-term outcomes. UR - https://medinform.jmir.org/2024/1/e53535 UR - http://dx.doi.org/10.2196/53535 ID - info:doi/10.2196/53535 ER - TY - JOUR AU - Kernberg, Annessa AU - Gold, A. Jeffrey AU - Mohan, Vishnu PY - 2024/4/22 TI - Using ChatGPT-4 to Create Structured Medical Notes From Audio Recordings of Physician-Patient Encounters: Comparative Study JO - J Med Internet Res SP - e54419 VL - 26 KW - generative AI KW - generative artificial intelligence KW - ChatGPT KW - simulation KW - large language model KW - clinical documentation KW - quality KW - accuracy KW - reproducibility KW - publicly available KW - medical note KW - medical notes KW - generation KW - medical documentation KW - documentation KW - documentations KW - AI KW - artificial intelligence KW - transcript KW - transcripts KW - ChatGPT-4 N2 - Background: Medical documentation plays a crucial role in clinical practice, facilitating accurate patient management and communication among health care professionals. However, inaccuracies in medical notes can lead to miscommunication and diagnostic errors. Additionally, the demands of documentation contribute to physician burnout. Although intermediaries like medical scribes and speech recognition software have been used to ease this burden, they have limitations in terms of accuracy and addressing provider-specific metrics. The integration of ambient artificial intelligence (AI)?powered solutions offers a promising way to improve documentation while fitting seamlessly into existing workflows. Objective: This study aims to assess the accuracy and quality of Subjective, Objective, Assessment, and Plan (SOAP) notes generated by ChatGPT-4, an AI model, using established transcripts of History and Physical Examination as the gold standard. We seek to identify potential errors and evaluate the model?s performance across different categories. Methods: We conducted simulated patient-provider encounters representing various ambulatory specialties and transcribed the audio files. Key reportable elements were identified, and ChatGPT-4 was used to generate SOAP notes based on these transcripts. Three versions of each note were created and compared to the gold standard via chart review; errors generated from the comparison were categorized as omissions, incorrect information, or additions. We compared the accuracy of data elements across versions, transcript length, and data categories. Additionally, we assessed note quality using the Physician Documentation Quality Instrument (PDQI) scoring system. Results: Although ChatGPT-4 consistently generated SOAP-style notes, there were, on average, 23.6 errors per clinical case, with errors of omission (86%) being the most common, followed by addition errors (10.5%) and inclusion of incorrect facts (3.2%). There was significant variance between replicates of the same case, with only 52.9% of data elements reported correctly across all 3 replicates. The accuracy of data elements varied across cases, with the highest accuracy observed in the ?Objective? section. Consequently, the measure of note quality, assessed by PDQI, demonstrated intra- and intercase variance. Finally, the accuracy of ChatGPT-4 was inversely correlated to both the transcript length (P=.05) and the number of scorable data elements (P=.05). Conclusions: Our study reveals substantial variability in errors, accuracy, and note quality generated by ChatGPT-4. Errors were not limited to specific sections, and the inconsistency in error types across replicates complicated predictability. Transcript length and data complexity were inversely correlated with note accuracy, raising concerns about the model?s effectiveness in handling complex medical cases. The quality and reliability of clinical notes produced by ChatGPT-4 do not meet the standards required for clinical use. Although AI holds promise in health care, caution should be exercised before widespread adoption. Further research is needed to address accuracy, variability, and potential errors. ChatGPT-4, while valuable in various applications, should not be considered a safe alternative to human-generated clinical documentation at this time. UR - https://www.jmir.org/2024/1/e54419 UR - http://dx.doi.org/10.2196/54419 UR - http://www.ncbi.nlm.nih.gov/pubmed/38648636 ID - info:doi/10.2196/54419 ER - TY - JOUR AU - Romano, D. Joseph AU - Truong, Van AU - Kumar, Rachit AU - Venkatesan, Mythreye AU - Graham, E. Britney AU - Hao, Yun AU - Matsumoto, Nick AU - Li, Xi AU - Wang, Zhiping AU - Ritchie, D. Marylyn AU - Shen, Li AU - Moore, H. Jason PY - 2024/4/18 TI - The Alzheimer?s Knowledge Base: A Knowledge Graph for Alzheimer Disease Research JO - J Med Internet Res SP - e46777 VL - 26 KW - Alzheimer disease KW - knowledge graph KW - knowledge base KW - artificial intelligence KW - drug repurposing KW - drug discovery KW - open source KW - Alzheimer KW - etiology KW - heterogeneous graph KW - therapeutic targets KW - machine learning KW - therapeutic discovery N2 - Background: As global populations age and become susceptible to neurodegenerative illnesses, new therapies for Alzheimer disease (AD) are urgently needed. Existing data resources for drug discovery and repurposing fail to capture relationships central to the disease?s etiology and response to drugs. Objective: We designed the Alzheimer?s Knowledge Base (AlzKB) to alleviate this need by providing a comprehensive knowledge representation of AD etiology and candidate therapeutics. Methods: We designed the AlzKB as a large, heterogeneous graph knowledge base assembled using 22 diverse external data sources describing biological and pharmaceutical entities at different levels of organization (eg, chemicals, genes, anatomy, and diseases). AlzKB uses a Web Ontology Language 2 ontology to enforce semantic consistency and allow for ontological inference. We provide a public version of AlzKB and allow users to run and modify local versions of the knowledge base. Results: AlzKB is freely available on the web and currently contains 118,902 entities with 1,309,527 relationships between those entities. To demonstrate its value, we used graph data science and machine learning to (1) propose new therapeutic targets based on similarities of AD to Parkinson disease and (2) repurpose existing drugs that may treat AD. For each use case, AlzKB recovers known therapeutic associations while proposing biologically plausible new ones. Conclusions: AlzKB is a new, publicly available knowledge resource that enables researchers to discover complex translational associations for AD drug discovery. Through 2 use cases, we show that it is a valuable tool for proposing novel therapeutic hypotheses based on public biomedical knowledge. UR - https://www.jmir.org/2024/1/e46777 UR - http://dx.doi.org/10.2196/46777 UR - http://www.ncbi.nlm.nih.gov/pubmed/38635981 ID - info:doi/10.2196/46777 ER - TY - JOUR AU - Caruso, Rosario AU - Di Muzio, Marco AU - Di Simone, Emanuele AU - Dionisi, Sara AU - Magon, Arianna AU - Conte, Gianluca AU - Stievano, Alessandro AU - Girani, Emanuele AU - Boveri, Sara AU - Menicanti, Lorenzo AU - Dolansky, A. Mary PY - 2024/4/17 TI - Global Trends of Medical Misadventures Using International Classification of Diseases, Tenth Revision Cluster Y62-Y69 Comparing Pre?, Intra?, and Post?COVID-19 Pandemic Phases: Protocol for a Retrospective Analysis Using the TriNetX Platform JO - JMIR Res Protoc SP - e54838 VL - 13 KW - COVID-19 KW - curve-fitting analyses KW - health care quality KW - health care safety KW - International Classification of Diseases, Tenth Revision KW - ICD-10 KW - incidence rates KW - safety KW - TriNetX N2 - Background: The COVID-19 pandemic has sharpened the focus on health care safety and quality, underscoring the importance of using standardized metrics such as the International Classification of Diseases, Tenth Revision (ICD-10). In this regard, the ICD-10 cluster Y62-Y69 serves as a proxy assessment of safety and quality in health care systems, allowing researchers to evaluate medical misadventures. Thus far, extensive research and reports support the need for more attention to safety and quality in health care. The study aims to leverage the pandemic?s unique challenges to explore health care safety and quality trends during prepandemic, intrapandemic, and postpandemic phases, using the ICD-10 cluster Y62-Y69 as a key tool for their evaluation. Objective: This research aims to perform a comprehensive retrospective analysis of incidence rates associated with ICD-10 cluster Y62-Y69, capturing both linear and nonlinear trends across prepandemic, intrapandemic, and postpandemic phases over an 8-year span. Therefore, it seeks to understand how these trends inform health care safety and quality improvements, policy, and future research. Methods: This study uses the extensive data available through the TriNetX platform, using an observational, retrospective design and applying curve-fitting analyses and quadratic models to comprehend the relationships between incidence rates over an 8-year span (from 2015 to 2023). These techniques will enable the identification of nuanced trends in the data, facilitating a deeper understanding of the impacts of the COVID-19 pandemic on medical misadventures. The anticipated results aim to outline complex patterns in health care safety and quality during the COVID-19 pandemic, using global real-world data for robust and generalizable conclusions. This study will explore significant shifts in health care practices and outcomes, with a special focus on geographical variations and key clinical conditions in cardiovascular and oncological care, ensuring a comprehensive analysis of the pandemic?s impact across different regions and medical fields. Results: This study is currently in the data collection phase, with funding secured in November 2023 through the Ricerca Corrente scheme of the Italian Ministry of Health. Data collection via the TriNetX platform is anticipated to be completed in May 2024, covering an 8-year period from January 2015 to December 2023. This dataset spans pre-pandemic, intra-pandemic, and early post-pandemic phases, enabling a comprehensive analysis of trends in medical misadventures using the ICD-10 cluster Y62-Y69. The final analytics are anticipated to be completed by June 2024. The study's findings aim to provide actionable insights for enhancing healthcare safety and quality, reflecting on the pandemic's transformative impact on global healthcare systems. Conclusions: This study is anticipated to contribute significantly to health care safety and quality literature. It will provide actionable insights for health care professionals, policy makers, and researchers. It will highlight critical areas for intervention and funding to enhance health care safety and quality globally by examining the incidence rates of medical misadventures before, during, and after the pandemic. In addition, the use of global real-world data enhances the study?s strength by providing a practical view of health care safety and quality, paving the way for initiatives that are informed by data and tailored to specific contexts worldwide. This approach ensures the findings are applicable and actionable across different health care settings, contributing significantly to the global understanding and improvement of health care safety and quality. International Registered Report Identifier (IRRID): PRR1-10.2196/54838 UR - https://www.researchprotocols.org/2024/1/e54838 UR - http://dx.doi.org/10.2196/54838 UR - http://www.ncbi.nlm.nih.gov/pubmed/38630516 ID - info:doi/10.2196/54838 ER - TY - JOUR AU - Sivarajkumar, Sonish AU - Kelley, Mark AU - Samolyk-Mazzanti, Alyssa AU - Visweswaran, Shyam AU - Wang, Yanshan PY - 2024/4/8 TI - An Empirical Evaluation of Prompting Strategies for Large Language Models in Zero-Shot Clinical Natural Language Processing: Algorithm Development and Validation Study JO - JMIR Med Inform SP - e55318 VL - 12 KW - large language model KW - LLM KW - LLMs KW - natural language processing KW - NLP KW - in-context learning KW - prompt engineering KW - evaluation KW - zero-shot KW - few shot KW - prompting KW - GPT KW - language model KW - language KW - models KW - machine learning KW - clinical data KW - clinical information KW - extraction KW - BARD KW - Gemini KW - LLaMA-2 KW - heuristic KW - prompt KW - prompts KW - ensemble N2 - Background: Large language models (LLMs) have shown remarkable capabilities in natural language processing (NLP), especially in domains where labeled data are scarce or expensive, such as the clinical domain. However, to unlock the clinical knowledge hidden in these LLMs, we need to design effective prompts that can guide them to perform specific clinical NLP tasks without any task-specific training data. This is known as in-context learning, which is an art and science that requires understanding the strengths and weaknesses of different LLMs and prompt engineering approaches. Objective: The objective of this study is to assess the effectiveness of various prompt engineering techniques, including 2 newly introduced types?heuristic and ensemble prompts, for zero-shot and few-shot clinical information extraction using pretrained language models. Methods: This comprehensive experimental study evaluated different prompt types (simple prefix, simple cloze, chain of thought, anticipatory, heuristic, and ensemble) across 5 clinical NLP tasks: clinical sense disambiguation, biomedical evidence extraction, coreference resolution, medication status extraction, and medication attribute extraction. The performance of these prompts was assessed using 3 state-of-the-art language models: GPT-3.5 (OpenAI), Gemini (Google), and LLaMA-2 (Meta). The study contrasted zero-shot with few-shot prompting and explored the effectiveness of ensemble approaches. Results: The study revealed that task-specific prompt tailoring is vital for the high performance of LLMs for zero-shot clinical NLP. In clinical sense disambiguation, GPT-3.5 achieved an accuracy of 0.96 with heuristic prompts and 0.94 in biomedical evidence extraction. Heuristic prompts, alongside chain of thought prompts, were highly effective across tasks. Few-shot prompting improved performance in complex scenarios, and ensemble approaches capitalized on multiple prompt strengths. GPT-3.5 consistently outperformed Gemini and LLaMA-2 across tasks and prompt types. Conclusions: This study provides a rigorous evaluation of prompt engineering methodologies and introduces innovative techniques for clinical information extraction, demonstrating the potential of in-context learning in the clinical domain. These findings offer clear guidelines for future prompt-based clinical NLP research, facilitating engagement by non-NLP experts in clinical NLP advancements. To the best of our knowledge, this is one of the first works on the empirical evaluation of different prompt engineering approaches for clinical NLP in this era of generative artificial intelligence, and we hope that it will inspire and inform future research in this area. UR - https://medinform.jmir.org/2024/1/e55318 UR - http://dx.doi.org/10.2196/55318 UR - http://www.ncbi.nlm.nih.gov/pubmed/38587879 ID - info:doi/10.2196/55318 ER - TY - JOUR AU - Mugaanyi, Joseph AU - Cai, Liuying AU - Cheng, Sumei AU - Lu, Caide AU - Huang, Jing PY - 2024/4/5 TI - Evaluation of Large Language Model Performance and Reliability for Citations and References in Scholarly Writing: Cross-Disciplinary Study JO - J Med Internet Res SP - e52935 VL - 26 KW - large language models KW - accuracy KW - academic writing KW - AI KW - cross-disciplinary evaluation KW - scholarly writing KW - ChatGPT KW - GPT-3.5 KW - writing tool KW - scholarly KW - academic discourse KW - LLMs KW - machine learning algorithms KW - NLP KW - natural language processing KW - citations KW - references KW - natural science KW - humanities KW - chatbot KW - artificial intelligence N2 - Background: Large language models (LLMs) have gained prominence since the release of ChatGPT in late 2022. Objective: The aim of this study was to assess the accuracy of citations and references generated by ChatGPT (GPT-3.5) in two distinct academic domains: the natural sciences and humanities. Methods: Two researchers independently prompted ChatGPT to write an introduction section for a manuscript and include citations; they then evaluated the accuracy of the citations and Digital Object Identifiers (DOIs). Results were compared between the two disciplines. Results: Ten topics were included, including 5 in the natural sciences and 5 in the humanities. A total of 102 citations were generated, with 55 in the natural sciences and 47 in the humanities. Among these, 40 citations (72.7%) in the natural sciences and 36 citations (76.6%) in the humanities were confirmed to exist (P=.42). There were significant disparities found in DOI presence in the natural sciences (39/55, 70.9%) and the humanities (18/47, 38.3%), along with significant differences in accuracy between the two disciplines (18/55, 32.7% vs 4/47, 8.5%). DOI hallucination was more prevalent in the humanities (42/55, 89.4%). The Levenshtein distance was significantly higher in the humanities than in the natural sciences, reflecting the lower DOI accuracy. Conclusions: ChatGPT?s performance in generating citations and references varies across disciplines. Differences in DOI standards and disciplinary nuances contribute to performance variations. Researchers should consider the strengths and limitations of artificial intelligence writing tools with respect to citation accuracy. The use of domain-specific models may enhance accuracy. UR - https://www.jmir.org/2024/1/e52935 UR - http://dx.doi.org/10.2196/52935 UR - http://www.ncbi.nlm.nih.gov/pubmed/38578685 ID - info:doi/10.2196/52935 ER - TY - JOUR AU - McMurry, J. Andrew AU - Zipursky, R. Amy AU - Geva, Alon AU - Olson, L. Karen AU - Jones, R. James AU - Ignatov, Vladimir AU - Miller, A. Timothy AU - Mandl, D. Kenneth PY - 2024/4/4 TI - Moving Biosurveillance Beyond Coded Data Using AI for Symptom Detection From Physician Notes: Retrospective Cohort Study JO - J Med Internet Res SP - e53367 VL - 26 KW - natural language processing KW - COVID-19 KW - artificial intelligence KW - AI KW - public health, biosurveillance KW - surveillance KW - respiratory KW - infectious KW - pulmonary KW - SARS-CoV-2 KW - symptom KW - symptoms KW - detect KW - detection KW - pipeline KW - pipelines KW - clinical note KW - clinical notes KW - documentation KW - emergency KW - urgent KW - pediatric KW - pediatrics KW - paediatric KW - paediatrics KW - child KW - children KW - youth KW - adolescent KW - adolescents KW - teen KW - teens KW - teenager KW - teenagers KW - diagnose KW - diagnosis KW - diagnostic KW - diagnostics N2 - Background: Real-time surveillance of emerging infectious diseases necessitates a dynamically evolving, computable case definition, which frequently incorporates symptom-related criteria. For symptom detection, both population health monitoring platforms and research initiatives primarily depend on structured data extracted from electronic health records. Objective: This study sought to validate and test an artificial intelligence (AI)?based natural language processing (NLP) pipeline for detecting COVID-19 symptoms from physician notes in pediatric patients. We specifically study patients presenting to the emergency department (ED) who can be sentinel cases in an outbreak. Methods: Subjects in this retrospective cohort study are patients who are 21 years of age and younger, who presented to a pediatric ED at a large academic children?s hospital between March 1, 2020, and May 31, 2022. The ED notes for all patients were processed with an NLP pipeline tuned to detect the mention of 11 COVID-19 symptoms based on Centers for Disease Control and Prevention (CDC) criteria. For a gold standard, 3 subject matter experts labeled 226 ED notes and had strong agreement (F1-score=0.986; positive predictive value [PPV]=0.972; and sensitivity=1.0). F1-score, PPV, and sensitivity were used to compare the performance of both NLP and the International Classification of Diseases, 10th Revision (ICD-10) coding to the gold standard chart review. As a formative use case, variations in symptom patterns were measured across SARS-CoV-2 variant eras. Results: There were 85,678 ED encounters during the study period, including 4% (n=3420) with patients with COVID-19. NLP was more accurate at identifying encounters with patients that had any of the COVID-19 symptoms (F1-score=0.796) than ICD-10 codes (F1-score =0.451). NLP accuracy was higher for positive symptoms (sensitivity=0.930) than ICD-10 (sensitivity=0.300). However, ICD-10 accuracy was higher for negative symptoms (specificity=0.994) than NLP (specificity=0.917). Congestion or runny nose showed the highest accuracy difference (NLP: F1-score=0.828 and ICD-10: F1-score=0.042). For encounters with patients with COVID-19, prevalence estimates of each NLP symptom differed across variant eras. Patients with COVID-19 were more likely to have each NLP symptom detected than patients without this disease. Effect sizes (odds ratios) varied across pandemic eras. Conclusions: This study establishes the value of AI-based NLP as a highly effective tool for real-time COVID-19 symptom detection in pediatric patients, outperforming traditional ICD-10 methods. It also reveals the evolving nature of symptom prevalence across different virus variants, underscoring the need for dynamic, technology-driven approaches in infectious disease surveillance. UR - https://www.jmir.org/2024/1/e53367 UR - http://dx.doi.org/10.2196/53367 UR - http://www.ncbi.nlm.nih.gov/pubmed/38573752 ID - info:doi/10.2196/53367 ER - TY - JOUR AU - Raja, Hina AU - Munawar, Asim AU - Mylonas, Nikolaos AU - Delsoz, Mohammad AU - Madadi, Yeganeh AU - Elahi, Muhammad AU - Hassan, Amr AU - Abu Serhan, Hashem AU - Inam, Onur AU - Hernandez, Luis AU - Chen, Hao AU - Tran, Sang AU - Munir, Wuqaas AU - Abd-Alrazaq, Alaa AU - Yousefi, Siamak PY - 2024/3/22 TI - Automated Category and Trend Analysis of Scientific Articles on Ophthalmology Using Large Language Models: Development and Usability Study JO - JMIR Form Res SP - e52462 VL - 8 KW - Bidirectional and Auto-Regressive Transformers KW - BART KW - bidirectional encoder representations from transformers KW - BERT KW - ophthalmology KW - text classification KW - large language model KW - LLM KW - trend analysis N2 - Background: In this paper, we present an automated method for article classification, leveraging the power of large language models (LLMs). Objective: The aim of this study is to evaluate the applicability of various LLMs based on textual content of scientific ophthalmology papers. Methods: We developed a model based on natural language processing techniques, including advanced LLMs, to process and analyze the textual content of scientific papers. Specifically, we used zero-shot learning LLMs and compared Bidirectional and Auto-Regressive Transformers (BART) and its variants with Bidirectional Encoder Representations from Transformers (BERT) and its variants, such as distilBERT, SciBERT, PubmedBERT, and BioBERT. To evaluate the LLMs, we compiled a data set (retinal diseases [RenD] ) of 1000 ocular disease?related articles, which were expertly annotated by a panel of 6 specialists into 19 distinct categories. In addition to the classification of articles, we also performed analysis on different classified groups to find the patterns and trends in the field. Results: The classification results demonstrate the effectiveness of LLMs in categorizing a large number of ophthalmology papers without human intervention. The model achieved a mean accuracy of 0.86 and a mean F1-score of 0.85 based on the RenD data set. Conclusions: The proposed framework achieves notable improvements in both accuracy and efficiency. Its application in the domain of ophthalmology showcases its potential for knowledge organization and retrieval. We performed a trend analysis that enables researchers and clinicians to easily categorize and retrieve relevant papers, saving time and effort in literature review and information gathering as well as identification of emerging scientific trends within different disciplines. Moreover, the extendibility of the model to other scientific fields broadens its impact in facilitating research and trend analysis across diverse disciplines. UR - https://formative.jmir.org/2024/1/e52462 UR - http://dx.doi.org/10.2196/52462 UR - http://www.ncbi.nlm.nih.gov/pubmed/38517457 ID - info:doi/10.2196/52462 ER - TY - JOUR AU - Dally, Daniela AU - Amith, Muhammad AU - Mauldin, L. Rebecca AU - Thomas, Latisha AU - Dang, Yifang AU - Tao, Cui PY - 2024/3/13 TI - A Semantic Approach to Describe Social and Economic Characteristics That Impact Health Outcomes (Social Determinants of Health): Ontology Development Study JO - Online J Public Health Inform SP - e52845 VL - 16 KW - social determinants of health KW - ontology KW - semantics KW - knowledge representation N2 - Background: Social determinants of health (SDoH) have been described by the World Health Organization as the conditions in which individuals are born, live, work, and age. These conditions can be grouped into 3 interrelated levels known as macrolevel (societal), mesolevel (community), and microlevel (individual) determinants. The scope of SDoH expands beyond the biomedical level, and there remains a need to connect other areas such as economics, public policy, and social factors. Objective: Providing a computable artifact that can link health data to concepts involving the different levels of determinants may improve our understanding of the impact SDoH have on human populations. Modeling SDoH may help to reduce existing gaps in the literature through explicit links between the determinants and biological factors. This in turn can allow researchers and clinicians to make better sense of data and discover new knowledge through the use of semantic links. Methods: An experimental ontology was developed to represent knowledge of the social and economic characteristics of SDoH. Information from 27 literature sources was analyzed to gather concepts and encoded using Web Ontology Language, version 2 (OWL2) and Protégé. Four evaluators independently reviewed the ontology axioms using natural language translation. The analyses from the evaluations and selected terminologies from the Basic Formal Ontology were used to create a revised ontology with a broad spectrum of knowledge concepts ranging from the macrolevel to the microlevel determinants. Results: The literature search identified several topics of discussion for each determinant level. Publications for the macrolevel determinants centered around health policy, income inequality, welfare, and the environment. Articles relating to the mesolevel determinants discussed work, work conditions, psychosocial factors, socioeconomic position, outcomes, food, poverty, housing, and crime. Finally, sources found for the microlevel determinants examined gender, ethnicity, race, and behavior. Concepts were gathered from the literature and used to produce an ontology consisting of 383 classes, 109 object properties, and 748 logical axioms. A reasoning test revealed no inconsistent axioms. Conclusions: This ontology models heterogeneous social and economic concepts to represent aspects of SDoH. The scope of SDoH is expansive, and although the ontology is broad, it is still in its early stages. To our current understanding, this ontology represents the first attempt to concentrate on knowledge concepts that are currently not covered by existing ontologies. Future direction will include further expanding the ontology to link with other biomedical ontologies, including alignment for granular semantics. UR - https://ojphi.jmir.org/2024/1/e52845 UR - http://dx.doi.org/10.2196/52845 UR - http://www.ncbi.nlm.nih.gov/pubmed/38477963 ID - info:doi/10.2196/52845 ER - TY - JOUR AU - Chomutare, Taridzo AU - Lamproudis, Anastasios AU - Budrionis, Andrius AU - Svenning, Olsen Therese AU - Hind, Irene Lill AU - Ngo, Dinh Phuong AU - Mikalsen, Øyvind Karl AU - Dalianis, Hercules PY - 2024/3/12 TI - Improving Quality of ICD-10 (International Statistical Classification of Diseases, Tenth Revision) Coding Using AI: Protocol for a Crossover Randomized Controlled Trial JO - JMIR Res Protoc SP - e54593 VL - 13 KW - International Classification of Diseases, Tenth Revision KW - ICD-10 KW - International Classification of Diseases, Eleventh Revision KW - ICD-11 KW - Easy-ICD KW - clinical coding KW - artificial intelligence KW - machine learning KW - deep learning N2 - Background: Computer-assisted clinical coding (CAC) tools are designed to help clinical coders assign standardized codes, such as the ICD-10 (International Statistical Classification of Diseases, Tenth Revision), to clinical texts, such as discharge summaries. Maintaining the integrity of these standardized codes is important both for the functioning of health systems and for ensuring data used for secondary purposes are of high quality. Clinical coding is an error-prone cumbersome task, and the complexity of modern classification systems such as the ICD-11 (International Classification of Diseases, Eleventh Revision) presents significant barriers to implementation. To date, there have only been a few user studies; therefore, our understanding is still limited regarding the role CAC systems can play in reducing the burden of coding and improving the overall quality of coding. Objective: The objective of the user study is to generate both qualitative and quantitative data for measuring the usefulness of a CAC system, Easy-ICD, that was developed for recommending ICD-10 codes. Specifically, our goal is to assess whether our tool can reduce the burden on clinical coders and also improve coding quality. Methods: The user study is based on a crossover randomized controlled trial study design, where we measure the performance of clinical coders when they use our CAC tool versus when they do not. Performance is measured by the time it takes them to assign codes to both simple and complex clinical texts as well as the coding quality, that is, the accuracy of code assignment. Results: We expect the study to provide us with a measurement of the effectiveness of the CAC system compared to manual coding processes, both in terms of time use and coding quality. Positive outcomes from this study will imply that CAC tools hold the potential to reduce the burden on health care staff and will have major implications for the adoption of artificial intelligence?based CAC innovations to improve coding practice. Expected results to be published summer 2024. Conclusions: The planned user study promises a greater understanding of the impact CAC systems might have on clinical coding in real-life settings, especially with regard to coding time and quality. Further, the study may add new insights on how to meaningfully exploit current clinical text mining capabilities, with a view to reducing the burden on clinical coders, thus lowering the barriers and paving a more sustainable path to the adoption of modern coding systems, such as the new ICD-11. Trial Registration: clinicaltrials.gov NCT06286865; https://clinicaltrials.gov/study/NCT06286865 International Registered Report Identifier (IRRID): DERR1-10.2196/54593 UR - https://www.researchprotocols.org/2024/1/e54593 UR - http://dx.doi.org/10.2196/54593 UR - http://www.ncbi.nlm.nih.gov/pubmed/38470476 ID - info:doi/10.2196/54593 ER - TY - JOUR AU - Xu, Yicong AU - Zhou, Jingya AU - Li, Hongxia AU - Cai, Dong AU - Zhu, Huanbing AU - Pan, Shengdong PY - 2024/3/8 TI - Improvements in Neoplasm Classification in the International Classification of Diseases, Eleventh Revision: Systematic Comparative Study With the Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision JO - Interact J Med Res SP - e52296 VL - 13 KW - Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision KW - ICD-10 KW - ICD-10-CCM KW - ICD-11 KW - improvement KW - International Classification of Diseases, Eleventh Revision KW - International Classification of Diseases, Tenth Revision KW - International Classification of Diseases KW - neoplasm KW - transition N2 - Background: The International Classification of Diseases, Eleventh Revision (ICD-11) improved neoplasm classification. Objective: We aimed to study the alterations in the ICD-11 compared to the Chinese Clinical Modification of the International Classification of Diseases, Tenth Revision (ICD-10-CCM) for neoplasm classification and to provide evidence supporting the transition to the ICD-11. Methods: We downloaded public data files from the World Health Organization and the National Health Commission of the People?s Republic of China. The ICD-10-CCM neoplasm codes were manually recoded with the ICD-11 coding tool, and an ICD-10-CCM/ICD-11 mapping table was generated. The existing files and the ICD-10-CCM/ICD-11 mapping table were used to compare the coding, classification, and expression features of neoplasms between the ICD-10-CCM and ICD-11. Results: The ICD-11 coding structure for neoplasms has dramatically changed. It provides advantages in coding granularity, coding capacity, and expression flexibility. In total, 27.4% (207/755) of ICD-10 codes and 38% (1359/3576) of ICD-10-CCM codes underwent grouping changes, which was a significantly different change (?21=30.3; P<.001). Notably, 67.8% (2424/3576) of ICD-10-CCM codes could be fully represented by ICD-11 codes. Another 7% (252/3576) could be fully described by uniform resource identifiers. The ICD-11 had a significant difference in expression ability among the 4 ICD-10-CCM groups (?23=93.7; P<.001), as well as a considerable difference between the changed and unchanged groups (?21=74.7; P<.001). Expression ability negatively correlated with grouping changes (r=?.144; P<.001). In the ICD-10-CCM/ICD-11 mapping table, 60.5% (2164/3576) of codes were postcoordinated. The top 3 postcoordinated results were specific anatomy (1907/3576, 53.3%), histopathology (201/3576, 5.6%), and alternative severity 2 (70/3576, 2%). The expression ability of postcoordination was not fully reflected. Conclusions: The ICD-11 includes many improvements in neoplasm classification, especially the new coding system, improved expression ability, and good semantic interoperability. The transition to the ICD-11 will inevitably bring challenges for clinicians, coders, policy makers and IT technicians, and many preparations will be necessary. UR - https://www.i-jmr.org/2024/1/e52296 UR - http://dx.doi.org/10.2196/52296 UR - http://www.ncbi.nlm.nih.gov/pubmed/38457228 ID - info:doi/10.2196/52296 ER - TY - JOUR AU - Declerck, Jens AU - Kalra, Dipak AU - Vander Stichele, Robert AU - Coorevits, Pascal PY - 2024/3/6 TI - Frameworks, Dimensions, Definitions of Aspects, and Assessment Methods for the Appraisal of Quality of Health Data for Secondary Use: Comprehensive Overview of Reviews JO - JMIR Med Inform SP - e51560 VL - 12 KW - data quality KW - data quality dimensions KW - data quality assessment KW - secondary use KW - data quality framework KW - fit for purpose N2 - Background: Health care has not reached the full potential of the secondary use of health data because of?among other issues?concerns about the quality of the data being used. The shift toward digital health has led to an increase in the volume of health data. However, this increase in quantity has not been matched by a proportional improvement in the quality of health data. Objective: This review aims to offer a comprehensive overview of the existing frameworks for data quality dimensions and assessment methods for the secondary use of health data. In addition, it aims to consolidate the results into a unified framework. Methods: A review of reviews was conducted including reviews describing frameworks of data quality dimensions and their assessment methods, specifically from a secondary use perspective. Reviews were excluded if they were not related to the health care ecosystem, lacked relevant information related to our research objective, and were published in languages other than English. Results: A total of 22 reviews were included, comprising 22 frameworks, with 23 different terms for dimensions, and 62 definitions of dimensions. All dimensions were mapped toward the data quality framework of the European Institute for Innovation through Health Data. In total, 8 reviews mentioned 38 different assessment methods, pertaining to 31 definitions of the dimensions. Conclusions: The findings in this review revealed a lack of consensus in the literature regarding the terminology, definitions, and assessment methods for data quality dimensions. This creates ambiguity and difficulties in developing specific assessment methods. This study goes a step further by assigning all observed definitions to a consolidated framework of 9 data quality dimensions. UR - https://medinform.jmir.org/2024/1/e51560 UR - http://dx.doi.org/10.2196/51560 UR - http://www.ncbi.nlm.nih.gov/pubmed/38446534 ID - info:doi/10.2196/51560 ER - TY - JOUR AU - Abd-alrazaq, Alaa AU - Nashwan, J. Abdulqadir AU - Shah, Zubair AU - Abujaber, Ahmad AU - Alhuwail, Dari AU - Schneider, Jens AU - AlSaad, Rawan AU - Ali, Hazrat AU - Alomoush, Waleed AU - Ahmed, Arfan AU - Aziz, Sarah PY - 2024/3/5 TI - Machine Learning?Based Approach for Identifying Research Gaps: COVID-19 as a Case Study JO - JMIR Form Res SP - e49411 VL - 8 KW - research gaps KW - research gap KW - research topic KW - research topics KW - scientific literature KW - literature review KW - machine learning KW - COVID-19 KW - BERTopic KW - topic clustering KW - text analysis KW - BERT KW - NLP KW - natural language processing KW - review methods KW - review methodology KW - SARS-CoV-2 KW - coronavirus KW - COVID N2 - Background: Research gaps refer to unanswered questions in the existing body of knowledge, either due to a lack of studies or inconclusive results. Research gaps are essential starting points and motivation in scientific research. Traditional methods for identifying research gaps, such as literature reviews and expert opinions, can be time consuming, labor intensive, and prone to bias. They may also fall short when dealing with rapidly evolving or time-sensitive subjects. Thus, innovative scalable approaches are needed to identify research gaps, systematically assess the literature, and prioritize areas for further study in the topic of interest. Objective: In this paper, we propose a machine learning?based approach for identifying research gaps through the analysis of scientific literature. We used the COVID-19 pandemic as a case study. Methods: We conducted an analysis to identify research gaps in COVID-19 literature using the COVID-19 Open Research (CORD-19) data set, which comprises 1,121,433 papers related to the COVID-19 pandemic. Our approach is based on the BERTopic topic modeling technique, which leverages transformers and class-based term frequency-inverse document frequency to create dense clusters allowing for easily interpretable topics. Our BERTopic-based approach involves 3 stages: embedding documents, clustering documents (dimension reduction and clustering), and representing topics (generating candidates and maximizing candidate relevance). Results: After applying the study selection criteria, we included 33,206 abstracts in the analysis of this study. The final list of research gaps identified 21 different areas, which were grouped into 6 principal topics. These topics were: ?virus of COVID-19,? ?risk factors of COVID-19,? ?prevention of COVID-19,? ?treatment of COVID-19,? ?health care delivery during COVID-19,? ?and impact of COVID-19.? The most prominent topic, observed in over half of the analyzed studies, was ?the impact of COVID-19.? Conclusions: The proposed machine learning?based approach has the potential to identify research gaps in scientific literature. This study is not intended to replace individual literature research within a selected topic. Instead, it can serve as a guide to formulate precise literature search queries in specific areas associated with research questions that previous publications have earmarked for future exploration. Future research should leverage an up-to-date list of studies that are retrieved from the most common databases in the target area. When feasible, full texts or, at minimum, discussion sections should be analyzed rather than limiting their analysis to abstracts. Furthermore, future studies could evaluate more efficient modeling algorithms, especially those combining topic modeling with statistical uncertainty quantification, such as conformal prediction. UR - https://formative.jmir.org/2024/1/e49411 UR - http://dx.doi.org/10.2196/49411 UR - http://www.ncbi.nlm.nih.gov/pubmed/38441952 ID - info:doi/10.2196/49411 ER - TY - JOUR AU - Deiner, S. Michael AU - Deiner, A. Natalie AU - Hristidis, Vagelis AU - McLeod, D. Stephen AU - Doan, Thuy AU - Lietman, M. Thomas AU - Porco, C. Travis PY - 2024/3/1 TI - Use of Large Language Models to Assess the Likelihood of Epidemics From the Content of Tweets: Infodemiology Study JO - J Med Internet Res SP - e49139 VL - 26 KW - conjunctivitis KW - microblog KW - social media KW - generative large language model KW - Generative Pre-trained Transformers KW - GPT-3.5 KW - GPT-4 KW - epidemic detection KW - Twitter KW - X formerly known as Twitter KW - infectious eye disease N2 - Background: Previous work suggests that Google searches could be useful in identifying conjunctivitis epidemics. Content-based assessment of social media content may provide additional value in serving as early indicators of conjunctivitis and other systemic infectious diseases. Objective: We investigated whether large language models, specifically GPT-3.5 and GPT-4 (OpenAI), can provide probabilistic assessments of whether social media posts about conjunctivitis could indicate a regional outbreak. Methods: A total of 12,194 conjunctivitis-related tweets were obtained using a targeted Boolean search in multiple languages from India, Guam (United States), Martinique (France), the Philippines, American Samoa (United States), Fiji, Costa Rica, Haiti, and the Bahamas, covering the time frame from January 1, 2012, to March 13, 2023. By providing these tweets via prompts to GPT-3.5 and GPT-4, we obtained probabilistic assessments that were validated by 2 human raters. We then calculated Pearson correlations of these time series with tweet volume and the occurrence of known outbreaks in these 9 locations, with time series bootstrap used to compute CIs. Results: Probabilistic assessments derived from GPT-3.5 showed correlations of 0.60 (95% CI 0.47-0.70) and 0.53 (95% CI 0.40-0.65) with the 2 human raters, with higher results for GPT-4. The weekly averages of GPT-3.5 probabilities showed substantial correlations with weekly tweet volume for 44% (4/9) of the countries, with correlations ranging from 0.10 (95% CI 0.0-0.29) to 0.53 (95% CI 0.39-0.89), with larger correlations for GPT-4. More modest correlations were found for correlation with known epidemics, with substantial correlation only in American Samoa (0.40, 95% CI 0.16-0.81). Conclusions: These findings suggest that GPT prompting can efficiently assess the content of social media posts and indicate possible disease outbreaks to a degree of accuracy comparable to that of humans. Furthermore, we found that automated content analysis of tweets is related to tweet volume for conjunctivitis-related posts in some locations and to the occurrence of actual epidemics. Future work may improve the sensitivity and specificity of these methods for disease outbreak detection. UR - https://www.jmir.org/2024/1/e49139 UR - http://dx.doi.org/10.2196/49139 UR - http://www.ncbi.nlm.nih.gov/pubmed/38427404 ID - info:doi/10.2196/49139 ER - TY - JOUR AU - Laurentiev, John AU - Kim, Hyun Dae AU - Mahesri, Mufaddal AU - Wang, Kuan-Yuan AU - Bessette, G. Lily AU - York, Cassandra AU - Zakoul, Heidi AU - Lee, Been Su AU - Zhou, Li AU - Lin, Joshua Kueiyu PY - 2024/2/13 TI - Identifying Functional Status Impairment in People Living With Dementia Through Natural Language Processing of Clinical Documents: Cross-Sectional Study JO - J Med Internet Res SP - e47739 VL - 26 KW - activities of daily living KW - ADLs KW - clinical note KW - dementia KW - electronic health record KW - EHR KW - functional impairment KW - instrumental activities of daily living KW - iADLs KW - machine learning KW - natural language processing KW - NLP N2 - Background: Assessment of activities of daily living (ADLs) and instrumental ADLs (iADLs) is key to determining the severity of dementia and care needs among older adults. However, such information is often only documented in free-text clinical notes within the electronic health record and can be challenging to find. Objective: This study aims to develop and validate machine learning models to determine the status of ADL and iADL impairments based on clinical notes. Methods: This cross-sectional study leveraged electronic health record clinical notes from Mass General Brigham?s Research Patient Data Repository linked with Medicare fee-for-service claims data from 2007 to 2017 to identify individuals aged 65 years or older with at least 1 diagnosis of dementia. Notes for encounters both 180 days before and after the first date of dementia diagnosis were randomly sampled. Models were trained and validated using note sentences filtered by expert-curated keywords (filtered cohort) and further evaluated using unfiltered sentences (unfiltered cohort). The model?s performance was compared using area under the receiver operating characteristic curve and area under the precision-recall curve (AUPRC). Results: The study included 10,000 key-term?filtered sentences representing 441 people (n=283, 64.2% women; mean age 82.7, SD 7.9 years) and 1000 unfiltered sentences representing 80 people (n=56, 70% women; mean age 82.8, SD 7.5 years). Area under the receiver operating characteristic curve was high for the best-performing ADL and iADL models on both cohorts (>0.97). For ADL impairment identification, the random forest model achieved the best AUPRC (0.89, 95% CI 0.86-0.91) on the filtered cohort; the support vector machine model achieved the highest AUPRC (0.82, 95% CI 0.75-0.89) for the unfiltered cohort. For iADL impairment, the Bio+Clinical bidirectional encoder representations from transformers (BERT) model had the highest AUPRC (filtered: 0.76, 95% CI 0.68-0.82; unfiltered: 0.58, 95% CI 0.001-1.0). Compared with a keyword-search approach on the unfiltered cohort, machine learning reduced false-positive rates from 4.5% to 0.2% for ADL and 1.8% to 0.1% for iADL. Conclusions: In this study, we demonstrated the ability of machine learning models to accurately identify ADL and iADL impairment based on free-text clinical notes, which could be useful in determining the severity of dementia. UR - https://www.jmir.org/2024/1/e47739 UR - http://dx.doi.org/10.2196/47739 UR - http://www.ncbi.nlm.nih.gov/pubmed/38349732 ID - info:doi/10.2196/47739 ER - TY - JOUR AU - Yu, Peng AU - Fang, Changchang AU - Liu, Xiaolin AU - Fu, Wanying AU - Ling, Jitao AU - Yan, Zhiwei AU - Jiang, Yuan AU - Cao, Zhengyu AU - Wu, Maoxiong AU - Chen, Zhiteng AU - Zhu, Wengen AU - Zhang, Yuling AU - Abudukeremu, Ayiguli AU - Wang, Yue AU - Liu, Xiao AU - Wang, Jingfeng PY - 2024/2/9 TI - Performance of ChatGPT on the Chinese Postgraduate Examination for Clinical Medicine: Survey Study JO - JMIR Med Educ SP - e48514 VL - 10 KW - ChatGPT KW - Chinese Postgraduate Examination for Clinical Medicine KW - medical student KW - performance KW - artificial intelligence KW - medical care KW - qualitative feedback KW - medical education KW - clinical decision-making N2 - Background: ChatGPT, an artificial intelligence (AI) based on large-scale language models, has sparked interest in the field of health care. Nonetheless, the capabilities of AI in text comprehension and generation are constrained by the quality and volume of available training data for a specific language, and the performance of AI across different languages requires further investigation. While AI harbors substantial potential in medicine, it is imperative to tackle challenges such as the formulation of clinical care standards; facilitating cultural transitions in medical education and practice; and managing ethical issues including data privacy, consent, and bias. Objective: The study aimed to evaluate ChatGPT?s performance in processing Chinese Postgraduate Examination for Clinical Medicine questions, assess its clinical reasoning ability, investigate potential limitations with the Chinese language, and explore its potential as a valuable tool for medical professionals in the Chinese context. Methods: A data set of Chinese Postgraduate Examination for Clinical Medicine questions was used to assess the effectiveness of ChatGPT?s (version 3.5) medical knowledge in the Chinese language, which has a data set of 165 medical questions that were divided into three categories: (1) common questions (n=90) assessing basic medical knowledge, (2) case analysis questions (n=45) focusing on clinical decision-making through patient case evaluations, and (3) multichoice questions (n=30) requiring the selection of multiple correct answers. First of all, we assessed whether ChatGPT could meet the stringent cutoff score defined by the government agency, which requires a performance within the top 20% of candidates. Additionally, in our evaluation of ChatGPT?s performance on both original and encoded medical questions, 3 primary indicators were used: accuracy, concordance (which validates the answer), and the frequency of insights. Results: Our evaluation revealed that ChatGPT scored 153.5 out of 300 for original questions in Chinese, which signifies the minimum score set to ensure that at least 20% more candidates pass than the enrollment quota. However, ChatGPT had low accuracy in answering open-ended medical questions, with only 31.5% total accuracy. The accuracy for common questions, multichoice questions, and case analysis questions was 42%, 37%, and 17%, respectively. ChatGPT achieved a 90% concordance across all questions. Among correct responses, the concordance was 100%, significantly exceeding that of incorrect responses (n=57, 50%; P<.001). ChatGPT provided innovative insights for 80% (n=132) of all questions, with an average of 2.95 insights per accurate response. Conclusions: Although ChatGPT surpassed the passing threshold for the Chinese Postgraduate Examination for Clinical Medicine, its performance in answering open-ended medical questions was suboptimal. Nonetheless, ChatGPT exhibited high internal concordance and the ability to generate multiple insights in the Chinese language. Future research should investigate the language-based discrepancies in ChatGPT?s performance within the health care context. UR - https://mededu.jmir.org/2024/1/e48514 UR - http://dx.doi.org/10.2196/48514 UR - http://www.ncbi.nlm.nih.gov/pubmed/38335017 ID - info:doi/10.2196/48514 ER - TY - JOUR AU - Wang, Lei AU - Bi, Wenshuai AU - Zhao, Suling AU - Ma, Yinyao AU - Lv, Longting AU - Meng, Chenwei AU - Fu, Jingru AU - Lv, Hanlin PY - 2024/2/8 TI - Investigating the Impact of Prompt Engineering on the Performance of Large Language Models for Standardizing Obstetric Diagnosis Text: Comparative Study JO - JMIR Form Res SP - e53216 VL - 8 KW - obstetric data KW - similarity embedding KW - term standardization KW - large language models KW - LLMs N2 - Background: The accumulation of vast electronic medical records (EMRs) through medical informatization creates significant research value, particularly in obstetrics. Diagnostic standardization across different health care institutions and regions is vital for medical data analysis. Large language models (LLMs) have been extensively used for various medical tasks. Prompt engineering is key to use LLMs effectively. Objective: This study aims to evaluate and compare the performance of LLMs with various prompt engineering techniques on the task of standardizing obstetric diagnostic terminology using real-world obstetric data. Methods: The paper describes a 4-step approach used for mapping diagnoses in electronic medical records to the International Classification of Diseases, 10th revision, observation domain. First, similarity measures were used for mapping the diagnoses. Second, candidate mapping terms were collected based on similarity scores above a threshold, to be used as the training data set. For generating optimal mapping terms, we used two LLMs (ChatGLM2 and Qwen-14B-Chat [QWEN]) for zero-shot learning in step 3. Finally, a performance comparison was conducted by using 3 pretrained bidirectional encoder representations from transformers (BERTs), including BERT, whole word masking BERT, and momentum contrastive learning with BERT (MC-BERT), for unsupervised optimal mapping term generation in the fourth step. Results: LLMs and BERT demonstrated comparable performance at their respective optimal levels. LLMs showed clear advantages in terms of performance and efficiency in unsupervised settings. Interestingly, the performance of the LLMs varied significantly across different prompt engineering setups. For instance, when applying the self-consistency approach in QWEN, the F1-score improved by 5%, with precision increasing by 7.9%, outperforming the zero-shot method. Likewise, ChatGLM2 delivered similar rates of accurately generated responses. During the analysis, the BERT series served as a comparative model with comparable results. Among the 3 models, MC-BERT demonstrated the highest level of performance. However, the differences among the versions of BERT in this study were relatively insignificant. Conclusions: After applying LLMs to standardize diagnoses and designing 4 different prompts, we compared the results to those generated by the BERT model. Our findings indicate that QWEN prompts largely outperformed the other prompts, with precision comparable to that of the BERT model. These results demonstrate the potential of unsupervised approaches in improving the efficiency of aligning diagnostic terms in daily research and uncovering hidden information values in patient data. UR - https://formative.jmir.org/2024/1/e53216 UR - http://dx.doi.org/10.2196/53216 UR - http://www.ncbi.nlm.nih.gov/pubmed/38329787 ID - info:doi/10.2196/53216 ER - TY - JOUR AU - Ji, Jia AU - Hou, Yongshuai AU - Chen, Xinyu AU - Pan, Youcheng AU - Xiang, Yang PY - 2024/2/8 TI - Vision-Language Model for Generating Textual Descriptions From Clinical Images: Model Development and Validation Study JO - JMIR Form Res SP - e32690 VL - 8 KW - clinical image KW - radiology report generation KW - vision-language model KW - multistage fine-tuning KW - prior knowledge N2 - Background: The automatic generation of radiology reports, which seeks to create a free-text description from a clinical radiograph, is emerging as a pivotal intersection between clinical medicine and artificial intelligence. Leveraging natural language processing technologies can accelerate report creation, enhancing health care quality and standardization. However, most existing studies have not yet fully tapped into the combined potential of advanced language and vision models. Objective: The purpose of this study was to explore the integration of pretrained vision-language models into radiology report generation. This would enable the vision-language model to automatically convert clinical images into high-quality textual reports. Methods: In our research, we introduced a radiology report generation model named ClinicalBLIP, building upon the foundational InstructBLIP model and refining it using clinical image-to-text data sets. A multistage fine-tuning approach via low-rank adaptation was proposed to deepen the semantic comprehension of the visual encoder and the large language model for clinical imagery. Furthermore, prior knowledge was integrated through prompt learning to enhance the precision of the reports generated. Experiments were conducted on both the IU X-RAY and MIMIC-CXR data sets, with ClinicalBLIP compared to several leading methods. Results: Experimental results revealed that ClinicalBLIP obtained superior scores of 0.570/0.365 and 0.534/0.313 on the IU X-RAY/MIMIC-CXR test sets for the Metric for Evaluation of Translation with Explicit Ordering (METEOR) and the Recall-Oriented Understudy for Gisting Evaluation (ROUGE) evaluations, respectively. This performance notably surpasses that of existing state-of-the-art methods. Further evaluations confirmed the effectiveness of the multistage fine-tuning and the integration of prior information, leading to substantial improvements. Conclusions: The proposed ClinicalBLIP model demonstrated robustness and effectiveness in enhancing clinical radiology report generation, suggesting significant promise for real-world clinical applications. UR - https://formative.jmir.org/2024/1/e32690 UR - http://dx.doi.org/10.2196/32690 UR - http://www.ncbi.nlm.nih.gov/pubmed/38329788 ID - info:doi/10.2196/32690 ER - TY - JOUR AU - Koonce, Y. Taneya AU - Giuse, A. Dario AU - Williams, M. Annette AU - Blasingame, N. Mallory AU - Krump, A. Poppy AU - Su, Jing AU - Giuse, B. Nunzia PY - 2024/1/30 TI - Using a Natural Language Processing Approach to Support Rapid Knowledge Acquisition JO - JMIR Med Inform SP - e53516 VL - 12 KW - natural language processing KW - electronic health records KW - machine learning KW - data mining KW - knowledge management KW - NLP UR - https://medinform.jmir.org/2024/1/e53516 UR - http://dx.doi.org/10.2196/53516 UR - http://www.ncbi.nlm.nih.gov/pubmed/38289670 ID - info:doi/10.2196/53516 ER - TY - JOUR AU - Lee, You-Qian AU - Chen, Ching-Tai AU - Chen, Chien-Chang AU - Lee, Chung-Hong AU - Chen, Peitsz AU - Wu, Chi-Shin AU - Dai, Hong-Jie PY - 2024/1/25 TI - Unlocking the Secrets Behind Advanced Artificial Intelligence Language Models in Deidentifying Chinese-English Mixed Clinical Text: Development and Validation Study JO - J Med Internet Res SP - e48443 VL - 26 KW - code mixing KW - electronic health record KW - deidentification KW - pretrained language model KW - large language model KW - ChatGPT N2 - Background: The widespread use of electronic health records in the clinical and biomedical fields makes the removal of protected health information (PHI) essential to maintain privacy. However, a significant portion of information is recorded in unstructured textual forms, posing a challenge for deidentification. In multilingual countries, medical records could be written in a mixture of more than one language, referred to as code mixing. Most current clinical natural language processing techniques are designed for monolingual text, and there is a need to address the deidentification of code-mixed text. Objective: The aim of this study was to investigate the effectiveness and underlying mechanism of fine-tuned pretrained language models (PLMs) in identifying PHI in the code-mixed context. Additionally, we aimed to evaluate the potential of prompting large language models (LLMs) for recognizing PHI in a zero-shot manner. Methods: We compiled the first clinical code-mixed deidentification data set consisting of text written in Chinese and English. We explored the effectiveness of fine-tuned PLMs for recognizing PHI in code-mixed content, with a focus on whether PLMs exploit naming regularity and mention coverage to achieve superior performance, by probing the developed models? outputs to examine their decision-making process. Furthermore, we investigated the potential of prompt-based in-context learning of LLMs for recognizing PHI in code-mixed text. Results: The developed methods were evaluated on a code-mixed deidentification corpus of 1700 discharge summaries. We observed that different PHI types had preferences in their occurrences within the different types of language-mixed sentences, and PLMs could effectively recognize PHI by exploiting the learned name regularity. However, the models may exhibit suboptimal results when regularity is weak or mentions contain unknown words that the representations cannot generate well. We also found that the availability of code-mixed training instances is essential for the model?s performance. Furthermore, the LLM-based deidentification method was a feasible and appealing approach that can be controlled and enhanced through natural language prompts. Conclusions: The study contributes to understanding the underlying mechanism of PLMs in addressing the deidentification process in the code-mixed context and highlights the significance of incorporating code-mixed training instances into the model training phase. To support the advancement of research, we created a manipulated subset of the resynthesized data set available for research purposes. Based on the compiled data set, we found that the LLM-based deidentification method is a feasible approach, but carefully crafted prompts are essential to avoid unwanted output. However, the use of such methods in the hospital setting requires careful consideration of data security and privacy concerns. Further research could explore the augmentation of PLMs and LLMs with external knowledge to improve their strength in recognizing rare PHI. UR - https://www.jmir.org/2024/1/e48443 UR - http://dx.doi.org/10.2196/48443 UR - http://www.ncbi.nlm.nih.gov/pubmed/38271060 ID - info:doi/10.2196/48443 ER - TY - JOUR AU - Wang, Changyu AU - Liu, Siru AU - Li, Aiqing AU - Liu, Jialin PY - 2023/12/29 TI - Text Dialogue Analysis for Primary Screening of Mild Cognitive Impairment: Development and Validation Study JO - J Med Internet Res SP - e51501 VL - 25 KW - artificial intelligence KW - AI KW - AI models KW - ChatGPT KW - primary screening KW - mild cognitive impairment KW - standardization KW - prompt design KW - design KW - cognitive impairment KW - screening KW - model KW - clinician KW - diagnosis N2 - Background: Artificial intelligence models tailored to diagnose cognitive impairment have shown excellent results. However, it is unclear whether large linguistic models can rival specialized models by text alone. Objective: In this study, we explored the performance of ChatGPT for primary screening of mild cognitive impairment (MCI) and standardized the design steps and components of the prompts. Methods: We gathered a total of 174 participants from the DementiaBank screening and classified 70% of them into the training set and 30% of them into the test set. Only text dialogues were kept. Sentences were cleaned using a macro code, followed by a manual check. The prompt consisted of 5 main parts, including character setting, scoring system setting, indicator setting, output setting, and explanatory information setting. Three dimensions of variables from published studies were included: vocabulary (ie, word frequency and word ratio, phrase frequency and phrase ratio, and lexical complexity), syntax and grammar (ie, syntactic complexity and grammatical components), and semantics (ie, semantic density and semantic coherence). We used R 4.3.0. for the analysis of variables and diagnostic indicators. Results: Three additional indicators related to the severity of MCI were incorporated into the final prompt for the model. These indicators were effective in discriminating between MCI and cognitively normal participants: tip-of-the-tongue phenomenon (P<.001), difficulty with complex ideas (P<.001), and memory issues (P<.001). The final GPT-4 model achieved a sensitivity of 0.8636, a specificity of 0.9487, and an area under the curve of 0.9062 on the training set; on the test set, the sensitivity, specificity, and area under the curve reached 0.7727, 0.8333, and 0.8030, respectively. Conclusions: ChatGPT was effective in the primary screening of participants with possible MCI. Improved standardization of prompts by clinicians would also improve the performance of the model. It is important to note that ChatGPT is not a substitute for a clinician making a diagnosis. UR - https://www.jmir.org/2023/1/e51501 UR - http://dx.doi.org/10.2196/51501 UR - http://www.ncbi.nlm.nih.gov/pubmed/38157230 ID - info:doi/10.2196/51501 ER - TY - JOUR AU - Liao, Wenxiong AU - Liu, Zhengliang AU - Dai, Haixing AU - Xu, Shaochen AU - Wu, Zihao AU - Zhang, Yiyang AU - Huang, Xiaoke AU - Zhu, Dajiang AU - Cai, Hongmin AU - Li, Quanzheng AU - Liu, Tianming AU - Li, Xiang PY - 2023/12/28 TI - Differentiating ChatGPT-Generated and Human-Written Medical Texts: Quantitative Study JO - JMIR Med Educ SP - e48904 VL - 9 KW - ChatGPT KW - medical ethics KW - linguistic analysis KW - text classification KW - artificial intelligence KW - medical texts KW - machine learning N2 - Background: Large language models, such as ChatGPT, are capable of generating grammatically perfect and human-like text content, and a large number of ChatGPT-generated texts have appeared on the internet. However, medical texts, such as clinical notes and diagnoses, require rigorous validation, and erroneous medical content generated by ChatGPT could potentially lead to disinformation that poses significant harm to health care and the general public. Objective: This study is among the first on responsible artificial intelligence?generated content in medicine. We focus on analyzing the differences between medical texts written by human experts and those generated by ChatGPT and designing machine learning workflows to effectively detect and differentiate medical texts generated by ChatGPT. Methods: We first constructed a suite of data sets containing medical texts written by human experts and generated by ChatGPT. We analyzed the linguistic features of these 2 types of content and uncovered differences in vocabulary, parts-of-speech, dependency, sentiment, perplexity, and other aspects. Finally, we designed and implemented machine learning methods to detect medical text generated by ChatGPT. The data and code used in this paper are published on GitHub. Results: Medical texts written by humans were more concrete, more diverse, and typically contained more useful information, while medical texts generated by ChatGPT paid more attention to fluency and logic and usually expressed general terminologies rather than effective information specific to the context of the problem. A bidirectional encoder representations from transformers?based model effectively detected medical texts generated by ChatGPT, and the F1 score exceeded 95%. Conclusions: Although text generated by ChatGPT is grammatically perfect and human-like, the linguistic characteristics of generated medical texts were different from those written by human experts. Medical text generated by ChatGPT could be effectively detected by the proposed machine learning algorithms. This study provides a pathway toward trustworthy and accountable use of large language models in medicine. UR - https://mededu.jmir.org/2023/1/e48904 UR - http://dx.doi.org/10.2196/48904 UR - http://www.ncbi.nlm.nih.gov/pubmed/38153785 ID - info:doi/10.2196/48904 ER - TY - JOUR AU - Renner, Christopher AU - Reimer, Niklas AU - Christoph, Jan AU - Busch, Hauke AU - Metzger, Patrick AU - Boerries, Melanie AU - Ustjanzew, Arsenij AU - Boehm, Dominik AU - Unberath, Philipp PY - 2023/12/11 TI - Extending cBioPortal for Therapy Recommendation Documentation in Molecular Tumor Boards: Development and Usability Study JO - JMIR Med Inform SP - e50017 VL - 11 KW - molecular tumor board KW - documentation platform KW - usability evaluation KW - cBioPortal KW - precision medicine KW - genomics KW - health information interoperability KW - tumor KW - implementation KW - cancer KW - tool KW - platform KW - development KW - precision KW - use KW - user-centered N2 - Background: In molecular tumor boards (MTBs), patients with rare or advanced cancers are discussed by a multidisciplinary team of health care professionals. Software support for MTBs is lacking; in particular, tools for preparing and documenting MTB therapy recommendations need to be developed. Objective: We aimed to implement an extension to cBioPortal to provide a tool for the documentation of therapy recommendations from MTB sessions in a secure and standardized manner. The developed extension should be embedded in the patient view of cBioPortal to enable easy documentation during MTB sessions. The resulting architecture for storing therapy recommendations should be integrable into various hospital information systems. Methods: On the basis of a requirements analysis and technology analysis for authentication techniques, a prototype was developed and iteratively refined through a user-centered development process. In conclusion, the tool was evaluated via a usability evaluation, including interviews, structured questionnaires, and the System Usability Scale. Results: The patient view of cBioPortal was extended with a new tab that enables users to document MTB sessions and therapy recommendations. The role-based access control was expanded to allow for a finer distinction among the rights to view, edit, and delete data. The usability evaluation showed overall good usability and a System Usability Scale score of 83.57. Conclusions: This study demonstrates how cBioPortal can be extended to not only visualize MTB patient data but also be used as a documentation platform for therapy recommendations. UR - https://medinform.jmir.org/2023/1/e50017 UR - http://dx.doi.org/10.2196/50017 UR - http://www.ncbi.nlm.nih.gov/pubmed/38079196 ID - info:doi/10.2196/50017 ER - TY - JOUR AU - Buhr, Raphael Christoph AU - Smith, Harry AU - Huppertz, Tilman AU - Bahr-Hamm, Katharina AU - Matthias, Christoph AU - Blaikie, Andrew AU - Kelsey, Tom AU - Kuhn, Sebastian AU - Eckrich, Jonas PY - 2023/12/5 TI - ChatGPT Versus Consultants: Blinded Evaluation on Answering Otorhinolaryngology Case?Based Questions JO - JMIR Med Educ SP - e49183 VL - 9 KW - large language models KW - LLMs KW - LLM KW - artificial intelligence KW - AI KW - ChatGPT KW - otorhinolaryngology KW - ORL KW - digital health KW - chatbots KW - global health KW - low- and middle-income countries KW - telemedicine KW - telehealth KW - language model KW - chatbot N2 - Background: Large language models (LLMs), such as ChatGPT (Open AI), are increasingly used in medicine and supplement standard search engines as information sources. This leads to more ?consultations? of LLMs about personal medical symptoms. Objective: This study aims to evaluate ChatGPT?s performance in answering clinical case?based questions in otorhinolaryngology (ORL) in comparison to ORL consultants? answers. Methods: We used 41 case-based questions from established ORL study books and past German state examinations for doctors. The questions were answered by both ORL consultants and ChatGPT 3. ORL consultants rated all responses, except their own, on medical adequacy, conciseness, coherence, and comprehensibility using a 6-point Likert scale. They also identified (in a blinded setting) if the answer was created by an ORL consultant or ChatGPT. Additionally, the character count was compared. Due to the rapidly evolving pace of technology, a comparison between responses generated by ChatGPT 3 and ChatGPT 4 was included to give an insight into the evolving potential of LLMs. Results: Ratings in all categories were significantly higher for ORL consultants (P<.001). Although inferior to the scores of the ORL consultants, ChatGPT?s scores were relatively higher in semantic categories (conciseness, coherence, and comprehensibility) compared to medical adequacy. ORL consultants identified ChatGPT as the source correctly in 98.4% (121/123) of cases. ChatGPT?s answers had a significantly higher character count compared to ORL consultants (P<.001). Comparison between responses generated by ChatGPT 3 and ChatGPT 4 showed a slight improvement in medical accuracy as well as a better coherence of the answers provided. Contrarily, neither the conciseness (P=.06) nor the comprehensibility (P=.08) improved significantly despite the significant increase in the mean amount of characters by 52.5% (n= (1470-964)/964; P<.001). Conclusions: While ChatGPT provided longer answers to medical problems, medical adequacy and conciseness were significantly lower compared to ORL consultants? answers. LLMs have potential as augmentative tools for medical care, but their ?consultation? for medical problems carries a high risk of misinformation as their high semantic quality may mask contextual deficits. UR - https://mededu.jmir.org/2023/1/e49183 UR - http://dx.doi.org/10.2196/49183 UR - http://www.ncbi.nlm.nih.gov/pubmed/38051578 ID - info:doi/10.2196/49183 ER - TY - JOUR AU - Pirmani, Ashkan AU - De Brouwer, Edward AU - Geys, Lotte AU - Parciak, Tina AU - Moreau, Yves AU - Peeters, M. Liesbet PY - 2023/11/9 TI - The Journey of Data Within a Global Data Sharing Initiative: A Federated 3-Layer Data Analysis Pipeline to Scale Up Multiple Sclerosis Research JO - JMIR Med Inform SP - e48030 VL - 11 KW - data analysis pipeline KW - federated model sharing KW - real-world data KW - evidence-based decision-making KW - end-to-end pipeline KW - multiple sclerosis KW - data analysis KW - pipeline KW - data science KW - federated KW - neurology KW - brain KW - spine KW - spinal nervous system KW - neuroscience KW - data sharing KW - rare KW - low prevalence N2 - Background: Investigating low-prevalence diseases such as multiple sclerosis is challenging because of the rather small number of individuals affected by this disease and the scattering of real-world data across numerous data sources. These obstacles impair data integration, standardization, and analysis, which negatively impact the generation of significant meaningful clinical evidence. Objective: This study aims to present a comprehensive, research question?agnostic, multistakeholder-driven end-to-end data analysis pipeline that accommodates 3 prevalent data-sharing streams: individual data sharing, core data set sharing, and federated model sharing. Methods: A demand-driven methodology is employed for standardization, followed by 3 streams of data acquisition, a data quality enhancement process, a data integration procedure, and a concluding analysis stage to fulfill real-world data-sharing requirements. This pipeline?s effectiveness was demonstrated through its successful implementation in the COVID-19 and multiple sclerosis global data sharing initiative. Results: The global data sharing initiative yielded multiple scientific publications and provided extensive worldwide guidance for the community with multiple sclerosis. The pipeline facilitated gathering pertinent data from various sources, accommodating distinct sharing streams and assimilating them into a unified data set for subsequent statistical analysis or secure data examination. This pipeline contributed to the assembly of the largest data set of people with multiple sclerosis infected with COVID-19. Conclusions: The proposed data analysis pipeline exemplifies the potential of global stakeholder collaboration and underlines the significance of evidence-based decision-making. It serves as a paradigm for how data sharing initiatives can propel advancements in health care, emphasizing its adaptability and capacity to address diverse research inquiries. UR - https://medinform.jmir.org/2023/1/e48030 UR - http://dx.doi.org/10.2196/48030 UR - http://www.ncbi.nlm.nih.gov/pubmed/37943585 ID - info:doi/10.2196/48030 ER - TY - JOUR AU - Hirosawa, Takanobu AU - Kawamura, Ren AU - Harada, Yukinori AU - Mizuta, Kazuya AU - Tokumasu, Kazuki AU - Kaji, Yuki AU - Suzuki, Tomoharu AU - Shimizu, Taro PY - 2023/10/9 TI - ChatGPT-Generated Differential Diagnosis Lists for Complex Case?Derived Clinical Vignettes: Diagnostic Accuracy Evaluation JO - JMIR Med Inform SP - e48808 VL - 11 KW - artificial intelligence KW - AI chatbot KW - ChatGPT KW - large language models KW - clinical decision support KW - natural language processing KW - diagnostic excellence KW - language model KW - vignette KW - case study KW - diagnostic KW - accuracy KW - decision support KW - diagnosis N2 - Background: The diagnostic accuracy of differential diagnoses generated by artificial intelligence chatbots, including ChatGPT models, for complex clinical vignettes derived from general internal medicine (GIM) department case reports is unknown. Objective: This study aims to evaluate the accuracy of the differential diagnosis lists generated by both third-generation ChatGPT (ChatGPT-3.5) and fourth-generation ChatGPT (ChatGPT-4) by using case vignettes from case reports published by the Department of GIM of Dokkyo Medical University Hospital, Japan. Methods: We searched PubMed for case reports. Upon identification, physicians selected diagnostic cases, determined the final diagnosis, and displayed them into clinical vignettes. Physicians typed the determined text with the clinical vignettes in the ChatGPT-3.5 and ChatGPT-4 prompts to generate the top 10 differential diagnoses. The ChatGPT models were not specially trained or further reinforced for this task. Three GIM physicians from other medical institutions created differential diagnosis lists by reading the same clinical vignettes. We measured the rate of correct diagnosis within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and the top diagnosis. Results: In total, 52 case reports were analyzed. The rates of correct diagnosis by ChatGPT-4 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 83% (43/52), 81% (42/52), and 60% (31/52), respectively. The rates of correct diagnosis by ChatGPT-3.5 within the top 10 differential diagnosis lists, top 5 differential diagnosis lists, and top diagnosis were 73% (38/52), 65% (34/52), and 42% (22/52), respectively. The rates of correct diagnosis by ChatGPT-4 were comparable to those by physicians within the top 10 (43/52, 83% vs 39/52, 75%, respectively; P=.47) and within the top 5 (42/52, 81% vs 35/52, 67%, respectively; P=.18) differential diagnosis lists and top diagnosis (31/52, 60% vs 26/52, 50%, respectively; P=.43) although the difference was not significant. The ChatGPT models? diagnostic accuracy did not significantly vary based on open access status or the publication date (before 2011 vs 2022). Conclusions: This study demonstrates the potential diagnostic accuracy of differential diagnosis lists generated using ChatGPT-3.5 and ChatGPT-4 for complex clinical vignettes from case reports published by the GIM department. The rate of correct diagnoses within the top 10 and top 5 differential diagnosis lists generated by ChatGPT-4 exceeds 80%. Although derived from a limited data set of case reports from a single department, our findings highlight the potential utility of ChatGPT-4 as a supplementary tool for physicians, particularly for those affiliated with the GIM department. Further investigations should explore the diagnostic accuracy of ChatGPT by using distinct case materials beyond its training data. Such efforts will provide a comprehensive insight into the role of artificial intelligence in enhancing clinical decision-making. UR - https://medinform.jmir.org/2023/1/e48808 UR - http://dx.doi.org/10.2196/48808 UR - http://www.ncbi.nlm.nih.gov/pubmed/37812468 ID - info:doi/10.2196/48808 ER - TY - JOUR AU - Zhang, Ying AU - Li, Xiaoying AU - Liu, Yi AU - Li, Aihua AU - Yang, Xuemei AU - Tang, Xiaoli PY - 2023/10/5 TI - A Multilabel Text Classifier of Cancer Literature at the Publication Level: Methods Study of Medical Text Classification JO - JMIR Med Inform SP - e44892 VL - 11 KW - text classification KW - publication-level classifier KW - cancer literature KW - deep learning N2 - Background: Given the threat posed by cancer to human health, there is a rapid growth in the volume of data in the cancer field and interdisciplinary and collaborative research is becoming increasingly important for fine-grained classification. The low-resolution classifier of reported studies at the journal level fails to satisfy advanced searching demands, and a single label does not adequately characterize the literature originated from interdisciplinary research results. There is thus a need to establish a multilabel classifier with higher resolution to support literature retrieval for cancer research and reduce the burden of screening papers for clinical relevance. Objective: The primary objective of this research was to address the low-resolution issue of cancer literature classification due to the ambiguity of the existing journal-level classifier in order to support gaining high-relevance evidence for clinical consideration and all-sided results for literature retrieval. Methods: We trained a multilabel classifier with scalability for classifying the literature on cancer research directly at the publication level to assign proper content-derived labels based on the ?Bidirectional Encoder Representation from Transformers (BERT) + X? model and obtain the best option for X. First, a corpus of 70,599 cancer publications retrieved from the Dimensions database was divided into a training and a testing set in a ratio of 7:3. Second, using the classification terminology of International Cancer Research Partnership cancer types, we compared the performance of classifiers developed using BERT and 5 classical deep learning models, such as the text recurrent neural network (TextRNN) and FastText, followed by metrics analysis. Results: After comparing various combined deep learning models, we obtained a classifier based on the optimal combination ?BERT + TextRNN,? with a precision of 93.09%, a recall of 87.75%, and an F1-score of 90.34%. Moreover, we quantified the distinctive characteristics in the text structure and multilabel distribution in order to generalize the model to other fields with similar characteristics. Conclusions: The ?BERT + TextRNN? model was trained for high-resolution classification of cancer literature at the publication level to support accurate retrieval and academic statistics. The model automatically assigns 1 or more labels to each cancer paper, as required. Quantitative comparison verified that the ?BERT + TextRNN? model is the best fit for multilabel classification of cancer literature compared to other models. More data from diverse fields will be collected to testify the scalability and extensibility of the proposed model in the future. UR - https://medinform.jmir.org/2023/1/e44892 UR - http://dx.doi.org/10.2196/44892 UR - http://www.ncbi.nlm.nih.gov/pubmed/37796584 ID - info:doi/10.2196/44892 ER - TY - JOUR AU - Homburg, Maarten AU - Meijer, Eline AU - Berends, Matthijs AU - Kupers, Thijmen AU - Olde Hartman, Tim AU - Muris, Jean AU - de Schepper, Evelien AU - Velek, Premysl AU - Kuiper, Jeroen AU - Berger, Marjolein AU - Peters, Lilian PY - 2023/10/4 TI - A Natural Language Processing Model for COVID-19 Detection Based on Dutch General Practice Electronic Health Records by Using Bidirectional Encoder Representations From Transformers: Development and Validation Study JO - J Med Internet Res SP - e49944 VL - 25 KW - natural language processing KW - primary care KW - COVID-19 KW - EHR KW - electronic health records KW - public health KW - multidisciplinary KW - NLP KW - disease identification KW - BERT model KW - model development KW - prediction N2 - Background: Natural language processing (NLP) models such as bidirectional encoder representations from transformers (BERT) hold promise in revolutionizing disease identification from electronic health records (EHRs) by potentially enhancing efficiency and accuracy. However, their practical application in practice settings demands a comprehensive and multidisciplinary approach to development and validation. The COVID-19 pandemic highlighted challenges in disease identification due to limited testing availability and challenges in handling unstructured data. In the Netherlands, where general practitioners (GPs) serve as the first point of contact for health care, EHRs generated by these primary care providers contain a wealth of potentially valuable information. Nonetheless, the unstructured nature of free-text entries in EHRs poses challenges in identifying trends, detecting disease outbreaks, or accurately pinpointing COVID-19 cases. Objective: This study aims to develop and validate a BERT model for detecting COVID-19 consultations in general practice EHRs in the Netherlands. Methods: The BERT model was initially pretrained on Dutch language data and fine-tuned using a comprehensive EHR data set comprising confirmed COVID-19 GP consultations and non?COVID-19?related consultations. The data set was partitioned into a training and development set, and the model?s performance was evaluated on an independent test set that served as the primary measure of its effectiveness in COVID-19 detection. To validate the final model, its performance was assessed through 3 approaches. First, external validation was applied on an EHR data set from a different geographic region in the Netherlands. Second, validation was conducted using results of polymerase chain reaction (PCR) test data obtained from municipal health services. Lastly, correlation between predicted outcomes and COVID-19?related hospitalizations in the Netherlands was assessed, encompassing the period around the outbreak of the pandemic in the Netherlands, that is, the period before widespread testing. Results: The model development used 300,359 GP consultations. We developed a highly accurate model for COVID-19 consultations (accuracy 0.97, F1-score 0.90, precision 0.85, recall 0.85, specificity 0.99). External validations showed comparable high performance. Validation on PCR test data showed high recall but low precision and specificity. Validation using hospital data showed significant correlation between COVID-19 predictions of the model and COVID-19?related hospitalizations (F1-score 96.8; P<.001; R2=0.69). Most importantly, the model was able to predict COVID-19 cases weeks before the first confirmed case in the Netherlands. Conclusions: The developed BERT model was able to accurately identify COVID-19 cases among GP consultations even preceding confirmed cases. The validated efficacy of our BERT model highlights the potential of NLP models to identify disease outbreaks early, exemplifying the power of multidisciplinary efforts in harnessing technology for disease identification. Moreover, the implications of this study extend beyond COVID-19 and offer a blueprint for the early recognition of various illnesses, revealing that such models could revolutionize disease surveillance. UR - https://www.jmir.org/2023/1/e49944 UR - http://dx.doi.org/10.2196/49944 UR - http://www.ncbi.nlm.nih.gov/pubmed/37792444 ID - info:doi/10.2196/49944 ER - TY - JOUR AU - Liu, Leibo AU - Perez-Concha, Oscar AU - Nguyen, Anthony AU - Bennett, Vicki AU - Blake, Victoria AU - Gallego, Blanca AU - Jorm, Louisa PY - 2023/8/25 TI - Web-Based Application Based on Human-in-the-Loop Deep Learning for Deidentifying Free-Text Data in Electronic Medical Records: Development and Usability Study JO - Interact J Med Res SP - e46322 VL - 12 KW - web-based system KW - deidentification KW - electronic medical records KW - deep learning KW - narrative free text KW - human in the loop KW - free text KW - unstructured data KW - electronic health records KW - machine learning N2 - Background: The narrative free-text data in electronic medical records (EMRs) contain valuable clinical information for analysis and research to inform better patient care. However, the release of free text for secondary use is hindered by concerns surrounding personally identifiable information (PII), as protecting individuals' privacy is paramount. Therefore, it is necessary to deidentify free text to remove PII. Manual deidentification is a time-consuming and labor-intensive process. Numerous automated deidentification approaches and systems have been attempted to overcome this challenge over the past decade. Objective: We sought to develop an accurate, web-based system deidentifying free text (DEFT), which can be readily and easily adopted in real-world settings for deidentification of free text in EMRs. The system has several key features including a simple and task-focused web user interface, customized PII types, use of a state-of-the-art deep learning model for tagging PII from free text, preannotation by an interactive learning loop, rapid manual annotation with autosave, support for project management and team collaboration, user access control, and central data storage. Methods: DEFT comprises frontend and backend modules and communicates with central data storage through a filesystem path access. The frontend web user interface provides end users with a user-friendly workspace for managing and annotating free text. The backend module processes the requests from the frontend and performs relevant persistence operations. DEFT manages the deidentification workflow as a project, which can contain one or more data sets. Customized PII types and user access control can also be configured. The deep learning model is based on a Bidirectional Long Short-Term Memory-Conditional Random Field (BiLSTM-CRF) with RoBERTa as the word embedding layer. The interactive learning loop is further integrated into DEFT to speed up the deidentification process and increase its performance over time. Results: DEFT has many advantages over existing deidentification systems in terms of its support for project management, user access control, data management, and an interactive learning process. Experimental results from DEFT on the 2014 i2b2 data set obtained the highest performance compared to 5 benchmark models in terms of microaverage strict entity?level recall and F1-scores of 0.9563 and 0.9627, respectively. In a real-world use case of deidentifying clinical notes, extracted from 1 referral hospital in Sydney, New South Wales, Australia, DEFT achieved a high microaverage strict entity?level F1-score of 0.9507 on a corpus of 600 annotated clinical notes. Moreover, the manual annotation process with preannotation demonstrated a 43% increase in work efficiency compared to the process without preannotation. Conclusions: DEFT is designed for health domain researchers and data custodians to easily deidentify free text in EMRs. DEFT supports an interactive learning loop and end users with minimal technical knowledge can perform the deidentification work with only a shallow learning curve. UR - https://www.i-jmr.org/2023/1/e46322 UR - http://dx.doi.org/10.2196/46322 UR - http://www.ncbi.nlm.nih.gov/pubmed/37624624 ID - info:doi/10.2196/46322 ER - TY - JOUR AU - Golec, Marcin AU - Kamdar, Maulik AU - Barteit, Sandra PY - 2023/8/11 TI - Comprehensive Ontology of Fibroproliferative Diseases: Protocol for a Semantic Technology Study JO - JMIR Res Protoc SP - e48645 VL - 12 KW - fibroproliferative disease KW - fibrosis KW - fibrotic disease KW - ontology KW - OWL KW - semantic technology KW - Web Ontology Language N2 - Background: Fibroproliferative or fibrotic diseases (FDs), which represent a significant proportion of age-related pathologies and account for over 40% of mortality in developed nations, are often underrepresented in focused research. Typically, these conditions are studied individually, such as chronic obstructive pulmonary disease or idiopathic pulmonary fibrosis (IPF), rather than as a collective entity, thereby limiting the holistic understanding and development of effective treatments. To address this, we propose creating and publicizing a comprehensive fibroproliferative disease ontology (FDO) to unify the understanding of FDs. Objective: This paper aims to delineate the study protocol for the creation of the FDO, foster transparency and high quality standards during its development, and subsequently promote its use once it becomes publicly available. Methods: We aim to establish an ontology encapsulating the broad spectrum of FDs, constructed in the Web Ontology Language format using the Protégé ontology editor, adhering to ontology development life cycle principles. The modeling process will leverage Protégé in accordance with a methodologically defined process, involving targeted scoping reviews of MEDLINE and PubMed information, expert knowledge, and an ontology development process. A hybrid top-down and bottom-up strategy will guide the identification of core concepts and relations, conducted by a team of domain experts based on systematic iterations of scientific literature reviews. Results: The result will be an exhaustive FDO accommodating a wide variety of crucial biomedical concepts, augmented with synonyms, definitions, and references. The FDO aims to encapsulate diverse perspectives on the FD domain, including those of clinicians, health informaticians, medical researchers, and public health experts. Conclusions: The FDO is expected to stimulate broader and more in-depth FD research by enabling reasoning, inference, and the identification of relationships between concepts for application in multiple contexts, such as developing specialized software, fostering research communities, and enhancing domain comprehension. A common vocabulary and understanding of relationships among medical professionals could potentially expedite scientific progress and the discovery of innovative solutions. The publicly available FDO will form the foundation for future research, technological advancements, and public health initiatives. International Registered Report Identifier (IRRID): PRR1-10.2196/48645 UR - https://www.researchprotocols.org/2023/1/e48645 UR - http://dx.doi.org/10.2196/48645 UR - http://www.ncbi.nlm.nih.gov/pubmed/37566458 ID - info:doi/10.2196/48645 ER - TY - JOUR AU - Jaiswal, Aman AU - Katz, Alan AU - Nesca, Marcello AU - Milios, Evangelos PY - 2023/8/9 TI - Identifying Risk Factors Associated With Lower Back Pain in Electronic Medical Record Free Text: Deep Learning Approach Using Clinical Note Annotations JO - JMIR Med Inform SP - e45105 VL - 11 KW - machine learning KW - lower back pain KW - natural language processing KW - semantic textual similarity KW - electronic medical records KW - risk factors KW - deep learning N2 - Background: Lower back pain is a common weakening condition that affects a large population. It is a leading cause of disability and lost productivity, and the associated medical costs and lost wages place a substantial burden on individuals and society. Recent advances in artificial intelligence and natural language processing have opened new opportunities for the identification and management of risk factors for lower back pain. In this paper, we propose and train a deep learning model on a data set of clinical notes that have been annotated with relevant risk factors, and we evaluate the model?s performance in identifying risk factors in new clinical notes. Objective: The primary objective is to develop a novel deep learning approach to detect risk factors for underlying disease in patients presenting with lower back pain in clinical encounter notes. The secondary objective is to propose solutions to potential challenges of using deep learning and natural language processing techniques for identifying risk factors in electronic medical record free text and make practical recommendations for future research in this area. Methods: We manually annotated clinical notes for the presence of six risk factors for severe underlying disease in patients presenting with lower back pain. Data were highly imbalanced, with only 12% (n=296) of the annotated notes having at least one risk factor. To address imbalanced data, a combination of semantic textual similarity and regular expressions was used to further capture notes for annotation. Further analysis was conducted to study the impact of downsampling, binary formulation of multi-label classification, and unsupervised pretraining on classification performance. Results: Of 2749 labeled clinical notes, 347 exhibited at least one risk factor, while 2402 exhibited none. The initial analysis shows that downsampling the training set to equalize the ratio of clinical notes with and without risk factors improved the macro?area under the receiver operating characteristic curve (AUROC) by 2%. The Bidirectional Encoder Representations from Transformers (BERT) model improved the macro-AUROC by 15% over the traditional machine learning baseline. In experiment 2, the proposed BERT?convolutional neural network (CNN) model for longer texts improved (4% macro-AUROC) over the BERT baseline, and the multitask models are more stable for minority classes. In experiment 3, domain adaptation of BERTCNN using masked language modeling improved the macro-AUROC by 2%. Conclusions: Primary care clinical notes are likely to require manipulation to perform meaningful free-text analysis. The application of BERT models for multi-label classification on downsampled annotated clinical notes is useful in detecting risk factors suggesting an indication for imaging for patients with lower back pain. UR - https://medinform.jmir.org/2023/1/e45105 UR - http://dx.doi.org/10.2196/45105 ID - info:doi/10.2196/45105 ER - TY - JOUR AU - Ahmadi, Najia AU - Zoch, Michele AU - Kelbert, Patricia AU - Noll, Richard AU - Schaaf, Jannik AU - Wolfien, Markus AU - Sedlmayr, Martin PY - 2023/8/3 TI - Methods Used in the Development of Common Data Models for Health Data: Scoping Review JO - JMIR Med Inform SP - e45116 VL - 11 KW - common data model KW - common data elements KW - health data KW - electronic health record KW - Observational Medical Outcomes Partnership KW - stakeholder involvement KW - Data harmonisation KW - Interoperability KW - Standardized Data Repositories KW - Suggestive Development Process KW - Healthcare KW - Medical Informatics KW - N2 - Background: Common data models (CDMs) are essential tools for data harmonization, which can lead to significant improvements in the health domain. CDMs unite data from disparate sources and ease collaborations across institutions, resulting in the generation of large standardized data repositories across different entities. An overview of existing CDMs and methods used to develop these data sets may assist in the development process of future models for the health domain, such as for decision support systems. Objective: This scoping review investigates methods used in the development of CDMs for health data. We aim to provide a broad overview of approaches and guidelines that are used in the development of CDMs (ie, common data elements or common data sets) for different health domains on an international level. Methods: This scoping review followed the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist. We conducted the literature search in prominent databases, namely, PubMed, Web of Science, Science Direct, and Scopus, starting from January 2000 until March 2022. We identified and screened 1309 articles. The included articles were evaluated based on the type of adopted method, which was used in the conception, users? needs collection, implementation, and evaluation phases of CDMs, and whether stakeholders (such as medical experts, patients? representatives, and IT staff) were involved during the process. Moreover, the models were grouped into iterative or linear types based on the imperativeness of the stages during development. Results: We finally identified 59 articles that fit our eligibility criteria. Of these articles, 45 specifically focused on common medical conditions, 10 focused on rare medical conditions, and the remaining 4 focused on both conditions. The development process usually involved stakeholders but in different ways (eg, working group meetings, Delphi approaches, interviews, and questionnaires). Twenty-two models followed an iterative process. Conclusions: The included articles showed the diversity of methods used to develop a CDM in different domains of health. We highlight the need for more specialized CDM development methods in the health domain and propose a suggestive development process that might ease the development of CDMs in the health domain in the future. UR - https://medinform.jmir.org/2023/1/e45116 UR - http://dx.doi.org/10.2196/45116 UR - http://www.ncbi.nlm.nih.gov/pubmed/37535410 ID - info:doi/10.2196/45116 ER - TY - JOUR AU - Lu, Qi Kevin Jia AU - Meaney, Christopher AU - Guo, Elaine AU - Leung, Fok-Han PY - 2023/7/27 TI - Evaluating the Applicability of Existing Lexicon-Based Sentiment Analysis Techniques on Family Medicine Resident Feedback Field Notes: Retrospective Cohort Study JO - JMIR Med Educ SP - e41953 VL - 9 KW - medical education KW - medical resident KW - feedback KW - field note KW - text mining KW - data mining KW - sentiment analysis KW - lexicon KW - lexical KW - dictionary KW - dictionaries KW - vocabulary KW - resident KW - medical student KW - medical trainee KW - residency KW - utility KW - feasibility N2 - Background: Field notes, a form for resident-preceptor clinical encounter feedback, are widely adopted across Canadian medical residency training programs for documenting residents? performance. This process generates a sizeable cumulative collection of feedback text, which is difficult for medical education faculty to navigate. As sentiment analysis is a subfield of text mining that can efficiently synthesize the polarity of a text collection, sentiment analysis may serve as an innovative solution. Objective: This study aimed to examine the feasibility and utility of sentiment analysis using 3 popular sentiment lexicons on medical resident field notes. Methods: We used a retrospective cohort design, curating text data from University of Toronto medical resident field notes gathered over 2 years (from July 2019 to June 2021). Lexicon-based sentiment analysis was applied using 3 standardized dictionaries, modified by removing ambiguous words as determined by a medical subject matter expert. Our modified lexicons assigned words from the text data a sentiment score, and we aggregated the word-level scores to a document-level polarity score. Agreement between dictionaries was assessed, and the document-level polarity was correlated with the overall preceptor rating of the clinical encounter under assessment. Results: Across the 3 original dictionaries, approximately a third of labeled words in our field note corpus were deemed ambiguous and were removed to create modified dictionaries. Across the 3 modified dictionaries, the mean sentiment for the ?Strengths? section of the field notes was mildly positive, while it was slightly less positive in the ?Areas of Improvement? section. We observed reasonable agreement between dictionaries for sentiment scores in both field note sections. Overall, the proportion of positively labeled documents increased with the overall preceptor rating, and the proportion of negatively labeled documents decreased with the overall preceptor rating. Conclusions: Applying sentiment analysis to systematically analyze field notes is feasible. However, the applicability of existing lexicons is limited in the medical setting, even after the removal of ambiguous words. Limited applicability warrants the need to generate new dictionaries specific to the medical education context. Additionally, aspect-based sentiment analysis may be applied to navigate the more nuanced structure of texts when identifying sentiments. Ultimately, this will allow for more robust inferences to discover opportunities for improving resident teaching curriculums. UR - https://mededu.jmir.org/2023/1/e41953 UR - http://dx.doi.org/10.2196/41953 UR - http://www.ncbi.nlm.nih.gov/pubmed/37498660 ID - info:doi/10.2196/41953 ER - TY - JOUR AU - Calvo-Cidoncha, Elena AU - Verdinelli, Julián AU - González-Bueno, Javier AU - López-Soto, Alfonso AU - Camacho Hernando, Concepción AU - Pastor-Duran, Xavier AU - Codina-Jané, Carles AU - Lozano-Rubí, Raimundo PY - 2023/7/10 TI - An Ontology-Based Approach to Improving Medication Appropriateness in Older Patients: Algorithm Development and Validation Study JO - JMIR Med Inform SP - e45850 VL - 11 KW - biological ontologies KW - decision support systems KW - inappropriate prescribing KW - elderly KW - medication regimen complexity KW - anticholinergic drug burden KW - trigger tool KW - clinical KW - ontologies KW - pharmacy KW - medication KW - decision support KW - pharmaceutic KW - pharmacology KW - chronic condition KW - chronic disease KW - domain KW - adverse event KW - ontology-based KW - alert N2 - Background: Inappropriate medication in older patients with multimorbidity results in a greater risk of adverse drug events. Clinical decision support systems (CDSSs) are intended to improve medication appropriateness. One approach to improving CDSSs is to use ontologies instead of relational databases. Previously, we developed OntoPharma?an ontology-based CDSS for reducing medication prescribing errors. Objective: The primary aim was to model a domain for improving medication appropriateness in older patients (chronic patient domain). The secondary aim was to implement the version of OntoPharma containing the chronic patient domain in a hospital setting. Methods: A 4-step process was proposed. The first step was defining the domain scope. The chronic patient domain focused on improving medication appropriateness in older patients. A group of experts selected the following three use cases: medication regimen complexity, anticholinergic and sedative drug burden, and the presence of triggers for identifying possible adverse events. The second step was domain model representation. The implementation was conducted by medical informatics specialists and clinical pharmacists using Protégé-OWL (Stanford Center for Biomedical Informatics Research). The third step was OntoPharma-driven alert module adaptation. We reused the existing framework based on SPARQL to query ontologies. The fourth step was implementing the version of OntoPharma containing the chronic patient domain in a hospital setting. Alerts generated from July to September 2022 were analyzed. Results: We proposed 6 new classes and 5 new properties, introducing the necessary changes in the ontologies previously created. An alert is shown if the Medication Regimen Complexity Index is ?40, if the Drug Burden Index is ?1, or if there is a trigger based on an abnormal laboratory value. A total of 364 alerts were generated for 107 patients; 154 (42.3%) alerts were accepted. Conclusions: We proposed an ontology-based approach to provide support for improving medication appropriateness in older patients with multimorbidity in a scalable, sustainable, and reusable way. The chronic patient domain was built based on our previous research, reusing the existing framework. OntoPharma has been implemented in clinical practice and generates alerts, considering the following use cases: medication regimen complexity, anticholinergic and sedative drug burden, and the presence of triggers for identifying possible adverse events. UR - https://medinform.jmir.org/2023/1/e45850 UR - http://dx.doi.org/10.2196/45850 ID - info:doi/10.2196/45850 ER - TY - JOUR AU - Nishiyama, Tomohiro AU - Yada, Shuntaro AU - Wakamiya, Shoko AU - Hori, Satoko AU - Aramaki, Eiji PY - 2023/5/3 TI - Transferability Based on Drug Structure Similarity in the Automatic Classification of Noncompliant Drug Use on Social Media: Natural Language Processing Approach JO - J Med Internet Res SP - e44870 VL - 25 KW - data mining KW - machine learning KW - medication noncompliance KW - natural language processing KW - pharmacovigilance KW - transfer learning KW - text classification N2 - Background: Medication noncompliance is a critical issue because of the increased number of drugs sold on the web. Web-based drug distribution is difficult to control, causing problems such as drug noncompliance and abuse. The existing medication compliance surveys lack completeness because it is impossible to cover patients who do not go to the hospital or provide accurate information to their doctors, so a social media?based approach is being explored to collect information about drug use. Social media data, which includes information on drug usage by users, can be used to detect drug abuse and medication compliance in patients. Objective: This study aimed to assess how the structural similarity of drugs affects the efficiency of machine learning models for text classification of drug noncompliance. Methods: This study analyzed 22,022 tweets about 20 different drugs. The tweets were labeled as either noncompliant use or mention, noncompliant sales, general use, or general mention. The study compares 2 methods for training machine learning models for text classification: single-sub-corpus transfer learning, in which a model is trained on tweets about a single drug and then tested on tweets about other drugs, and multi-sub-corpus incremental learning, in which models are trained on tweets about drugs in order of their structural similarity. The performance of a machine learning model trained on a single subcorpus (a data set of tweets about a specific category of drugs) was compared to the performance of a model trained on multiple subcorpora (data sets of tweets about multiple categories of drugs). Results: The results showed that the performance of the model trained on a single subcorpus varied depending on the specific drug used for training. The Tanimoto similarity (a measure of the structural similarity between compounds) was weakly correlated with the classification results. The model trained by transfer learning a corpus of drugs with close structural similarity performed better than the model trained by randomly adding a subcorpus when the number of subcorpora was small. Conclusions: The results suggest that structural similarity improves the classification performance of messages about unknown drugs if the drugs in the training corpus are few. On the other hand, this indicates that there is little need to consider the influence of the Tanimoto structural similarity if a sufficient variety of drugs are ensured. UR - https://www.jmir.org/2023/1/e44870 UR - http://dx.doi.org/10.2196/44870 UR - http://www.ncbi.nlm.nih.gov/pubmed/37133915 ID - info:doi/10.2196/44870 ER - TY - JOUR AU - Karapetian, Karina AU - Jeon, Min Soo AU - Kwon, Jin-Won AU - Suh, Young-Kyoon PY - 2023/3/8 TI - Supervised Relation Extraction Between Suicide-Related Entities and Drugs: Development and Usability Study of an Annotated PubMed Corpus JO - J Med Internet Res SP - e41100 VL - 25 KW - suicide KW - adverse drug events KW - information extraction KW - relation classification KW - bidirectional encoder representations from transformers KW - pharmacovigilance KW - natural language processing KW - PubMed KW - corpus KW - language model N2 - Background: Drug-induced suicide has been debated as a crucial issue in both clinical and public health research. Published research articles contain valuable data on the drugs associated with suicidal adverse events. An automated process that extracts such information and rapidly detects drugs related to suicide risk is essential but has not been well established. Moreover, few data sets are available for training and validating classification models on drug-induced suicide. Objective: This study aimed to build a corpus of drug-suicide relations containing annotated entities for drugs, suicidal adverse events, and their relations. To confirm the effectiveness of the drug-suicide relation corpus, we evaluated the performance of a relation classification model using the corpus in conjunction with various embeddings. Methods: We collected the abstracts and titles of research articles associated with drugs and suicide from PubMed and manually annotated them along with their relations at the sentence level (adverse drug events, treatment, suicide means, or miscellaneous). To reduce the manual annotation effort, we preliminarily selected sentences with a pretrained zero-shot classifier or sentences containing only drug and suicide keywords. We trained a relation classification model using various Bidirectional Encoder Representations from Transformer embeddings with the proposed corpus. We then compared the performances of the model with different Bidirectional Encoder Representations from Transformer?based embeddings and selected the most suitable embedding for our corpus. Results: Our corpus comprised 11,894 sentences extracted from the titles and abstracts of the PubMed research articles. Each sentence was annotated with drug and suicide entities and the relationship between these 2 entities (adverse drug events, treatment, means, and miscellaneous). All of the tested relation classification models that were fine-tuned on the corpus accurately detected sentences of suicidal adverse events regardless of their pretrained type and data set properties. Conclusions: To our knowledge, this is the first and most extensive corpus of drug-suicide relations. UR - https://www.jmir.org/2023/1/e41100 UR - http://dx.doi.org/10.2196/41100 UR - http://www.ncbi.nlm.nih.gov/pubmed/36884281 ID - info:doi/10.2196/41100 ER - TY - JOUR AU - Sezgin, Emre AU - Hussain, Syed-Amad AU - Rust, Steve AU - Huang, Yungui PY - 2023/3/7 TI - Extracting Medical Information From Free-Text and Unstructured Patient-Generated Health Data Using Natural Language Processing Methods: Feasibility Study With Real-world Data JO - JMIR Form Res SP - e43014 VL - 7 KW - patient-generated health data KW - natural language processing KW - named entity recognition KW - patient health records KW - text notes KW - voice KW - audio real-world data N2 - Background: Patient-generated health data (PGHD) captured via smart devices or digital health technologies can reflect an individual health journey. PGHD enables tracking and monitoring of personal health conditions, symptoms, and medications out of the clinic, which is crucial for self-care and shared clinical decisions. In addition to self-reported measures and structured PGHD (eg, self-screening, sensor-based biometric data), free-text and unstructured PGHD (eg, patient care note, medical diary) can provide a broader view of a patient?s journey and health condition. Natural language processing (NLP) is used to process and analyze unstructured data to create meaningful summaries and insights, showing promise to improve the utilization of PGHD. Objective: Our aim is to understand and demonstrate the feasibility of an NLP pipeline to extract medication and symptom information from real-world patient and caregiver data. Methods: We report a secondary data analysis, using a data set collected from 24 parents of children with special health care needs (CSHCN) who were recruited via a nonrandom sampling approach. Participants used a voice-interactive app for 2 weeks, generating free-text patient notes (audio transcription or text entry). We built an NLP pipeline using a zero-shot approach (adaptive to low-resource settings). We used named entity recognition (NER) and medical ontologies (RXNorm and SNOMED CT [Systematized Nomenclature of Medicine Clinical Terms]) to identify medication and symptoms. Sentence-level dependency parse trees and part-of-speech tags were used to extract additional entity information using the syntactic properties of a note. We assessed the data; evaluated the pipeline with the patient notes; and reported the precision, recall, and F1 scores. Results: In total, 87 patient notes are included (audio transcriptions n=78 and text entries n=9) from 24 parents who have at least one CSHCN. The participants were between the ages of 26 and 59 years. The majority were White (n=22, 92%), had more than one child (n=16, 67%), lived in Ohio (n=22, 92%), had mid- or upper-mid household income (n=15, 62.5%), and had higher level education (n=24, 58%). Out of 87 notes, 30 were drug and medication related, and 46 were symptom related. We captured medication instances (medication, unit, quantity, and date) and symptoms satisfactorily (precision >0.65, recall >0.77, F1>0.72). These results indicate the potential when using NER and dependency parsing through an NLP pipeline on information extraction from unstructured PGHD. Conclusions: The proposed NLP pipeline was found to be feasible for use with real-world unstructured PGHD to accomplish medication and symptom extraction. Unstructured PGHD can be leveraged to inform clinical decision-making, remote monitoring, and self-care including medical adherence and chronic disease management. With customizable information extraction methods using NER and medical ontologies, NLP models can feasibly extract a broad range of clinical information from unstructured PGHD in low-resource settings (eg, a limited number of patient notes or training data). UR - https://formative.jmir.org/2023/1/e43014 UR - http://dx.doi.org/10.2196/43014 UR - http://www.ncbi.nlm.nih.gov/pubmed/36881467 ID - info:doi/10.2196/43014 ER - TY - JOUR AU - Frei, Johann AU - Kramer, Frank PY - 2023/2/28 TI - German Medical Named Entity Recognition Model and Data Set Creation Using Machine Translation and Word Alignment: Algorithm Development and Validation JO - JMIR Form Res SP - e39077 VL - 7 KW - natural language processing KW - named entity recognition KW - information extraction N2 - Background: Data mining in the field of medical data analysis often needs to rely solely on the processing of unstructured data to retrieve relevant data. For German natural language processing, few open medical neural named entity recognition (NER) models have been published before this work. A major issue can be attributed to the lack of German training data. Objective: We developed a synthetic data set and a novel German medical NER model for public access to demonstrate the feasibility of our approach. In order to bypass legal restrictions due to potential data leaks through model analysis, we did not make use of internal, proprietary data sets, which is a frequent veto factor for data set publication. Methods: The underlying German data set was retrieved by translation and word alignment of a public English data set. The data set served as a foundation for model training and evaluation. For demonstration purposes, our NER model follows a simple network architecture that is designed for low computational requirements. Results: The obtained data set consisted of 8599 sentences including 30,233 annotations. The model achieved a class frequency?averaged F1 score of 0.82 on the test set after training across 7 different NER types. Artifacts in the synthesized data set with regard to translation and alignment induced by the proposed method were exposed. The annotation performance was evaluated on an external data set and measured in comparison with an existing baseline model that has been trained on a dedicated German data set in a traditional fashion. We discussed the drop in annotation performance on an external data set for our simple NER model. Our model is publicly available. Conclusions: We demonstrated the feasibility of obtaining a data set and training a German medical NER model by the exclusive use of public training data through our suggested method. The discussion on the limitations of our approach includes ways to further mitigate remaining problems in future work. UR - https://formative.jmir.org/2023/1/e39077 UR - http://dx.doi.org/10.2196/39077 UR - http://www.ncbi.nlm.nih.gov/pubmed/36853741 ID - info:doi/10.2196/39077 ER - TY - JOUR AU - Sotoodeh, Mani AU - Zhang, Wenhui AU - Simpson, L. Roy AU - Hertzberg, Stover Vicki AU - Ho, C. Joyce PY - 2023/2/23 TI - A Comprehensive and Improved Definition for Hospital-Acquired Pressure Injury Classification Based on Electronic Health Records: Comparative Study JO - JMIR Med Inform SP - e40672 VL - 11 KW - pressure ulcer KW - decubitus ulcer KW - electronic medical records KW - bedsore KW - nursing KW - data mining KW - electronic health record KW - EHR KW - nursing assessment KW - pressure ulcer care KW - pressure ulcer prevention KW - EHR data KW - EHR systems KW - nursing quality N2 - Background: Patients develop pressure injuries (PIs) in the hospital owing to low mobility, exposure to localized pressure, circulatory conditions, and other predisposing factors. Over 2.5 million Americans develop PIs annually. The Center for Medicare and Medicaid considers hospital-acquired PIs (HAPIs) as the most frequent preventable event, and they are the second most common claim in lawsuits. With the growing use of electronic health records (EHRs) in hospitals, an opportunity exists to build machine learning models to identify and predict HAPI rather than relying on occasional manual assessments by human experts. However, accurate computational models rely on high-quality HAPI data labels. Unfortunately, the different data sources within EHRs can provide conflicting information on HAPI occurrence in the same patient. Furthermore, the existing definitions of HAPI disagree with each other, even within the same patient population. The inconsistent criteria make it impossible to benchmark machine learning methods to predict HAPI. Objective: The objective of this project was threefold. We aimed to identify discrepancies in HAPI sources within EHRs, to develop a comprehensive definition for HAPI classification using data from all EHR sources, and to illustrate the importance of an improved HAPI definition. Methods: We assessed the congruence among HAPI occurrences documented in clinical notes, diagnosis codes, procedure codes, and chart events from the Medical Information Mart for Intensive Care III database. We analyzed the criteria used for the 3 existing HAPI definitions and their adherence to the regulatory guidelines. We proposed the Emory HAPI (EHAPI), which is an improved and more comprehensive HAPI definition. We then evaluated the importance of the labels in training a HAPI classification model using tree-based and sequential neural network classifiers. Results: We illustrate the complexity of defining HAPI, with <13% of hospital stays having at least 3 PI indications documented across 4 data sources. Although chart events were the most common indicator, it was the only PI documentation for >49% of the stays. We demonstrate a lack of congruence across existing HAPI definitions and EHAPI, with only 219 stays having a consensus positive label. Our analysis highlights the importance of our improved HAPI definition, with classifiers trained using our labels outperforming others on a small manually labeled set from nurse annotators and a consensus set in which all definitions agreed on the label. Conclusions: Standardized HAPI definitions are important for accurately assessing HAPI nursing quality metric and determining HAPI incidence for preventive measures. We demonstrate the complexity of defining an occurrence of HAPI, given the conflicting and incomplete EHR data. Our EHAPI definition has favorable properties, making it a suitable candidate for HAPI classification tasks. UR - https://medinform.jmir.org/2023/1/e40672 UR - http://dx.doi.org/10.2196/40672 UR - http://www.ncbi.nlm.nih.gov/pubmed/36649481 ID - info:doi/10.2196/40672 ER - TY - JOUR AU - Klein, Z. Ari AU - Kunatharaju, Shriya AU - O'Connor, Karen AU - Gonzalez-Hernandez, Graciela PY - 2023/2/9 TI - Pregex: Rule-Based Detection and Extraction of Twitter Data in Pregnancy JO - J Med Internet Res SP - e40569 VL - 25 KW - natural language processing KW - data mining KW - social media KW - pregnancy UR - https://www.jmir.org/2023/1/e40569 UR - http://dx.doi.org/10.2196/40569 UR - http://www.ncbi.nlm.nih.gov/pubmed/36757756 ID - info:doi/10.2196/40569 ER - TY - JOUR AU - Vuokko, Riikka AU - Vakkuri, Anne AU - Palojoki, Sari PY - 2023/2/6 TI - Systematized Nomenclature of Medicine?Clinical Terminology (SNOMED CT) Clinical Use Cases in the Context of Electronic Health Record Systems: Systematic Literature Review JO - JMIR Med Inform SP - e43750 VL - 11 KW - clinical KW - electronic health record KW - EHR KW - review method KW - literature review KW - SNOMED CT KW - Systematized Nomenclature for Medicine KW - use case KW - terminology KW - terminologies KW - SNOMED N2 - Background: The Systematized Medical Nomenclature for Medicine?Clinical Terminology (SNOMED CT) is a clinical terminology system that provides a standardized and scientifically validated way of representing clinical information captured by clinicians. It can be integrated into electronic health records (EHRs) to increase the possibilities for effective data use and ensure a better quality of documentation that supports continuity of care, thus enabling better quality in the care process. Even though SNOMED CT consists of extensively studied clinical terminology, previous research has repeatedly documented a lack of scientific evidence for SNOMED CT in the form of reported clinical use cases in electronic health record systems. Objective: The aim of this study was to explore evidence in previous literature reviews of clinical use cases of SNOMED CT integrated into EHR systems or other clinical applications during the last 5 years of continued development. The study sought to identify the main clinical use purposes, use phases, and key clinical benefits documented in SNOMED CT use cases. Methods: The Cochrane review protocol was applied for the study design. The application of the protocol was modified step-by-step to fit the research problem by first defining the search strategy, identifying the articles for the review by isolating the exclusion and inclusion criteria for assessing the search results, and lastly, evaluating and summarizing the review results. Results: In total, 17 research articles illustrating SNOMED CT clinical use cases were reviewed. The use purpose of SNOMED CT was documented in all the articles, with the terminology as a standard in EHR being the most common (8/17). The clinical use phase was documented in all the articles. The most common category of use phases was SNOMED CT in development (6/17). Core benefits achieved by applying SNOMED CT in a clinical context were identified by the researchers. These were related to terminology use outcomes, that is, to data quality in general or to enabling a consistent way of indexing, storing, retrieving, and aggregating clinical data (8/17). Additional benefits were linked to the productivity of coding or to advances in the quality and continuity of care. Conclusions: While the SNOMED CT use categories were well supported by previous research, this review demonstrates that further systematic research on clinical use cases is needed to promote the scalability of the review results. To achieve the best out-of-use case reports, more emphasis is suggested on describing the contextual factors, such as the electronic health care system and the use of previous frameworks to enable comparability of results. A lesson to be drawn from our study is that SNOMED CT is essential for structuring clinical data; however, research is needed to gather more evidence of how SNOMED CT benefits clinical care and patient safety. UR - https://medinform.jmir.org/2023/1/e43750 UR - http://dx.doi.org/10.2196/43750 UR - http://www.ncbi.nlm.nih.gov/pubmed/36745498 ID - info:doi/10.2196/43750 ER - TY - JOUR AU - Jahn, Franziska AU - Ammenwerth, Elske AU - Dornauer, Verena AU - Höffner, Konrad AU - Bindel, Michelle AU - Karopka, Thomas AU - Winter, Alfred PY - 2023/1/20 TI - A Linked Open Data?Based Terminology to Describe Libre/Free and Open-source Software: Incremental Development Study JO - JMIR Med Inform SP - e38861 VL - 11 KW - health informatics KW - ontology KW - free/libre open-source software KW - software applications KW - health IT KW - terminology N2 - Background: There is a variety of libre/free and open-source software (LIFOSS) products for medicine and health care. To support health care and IT professionals select an appropriate software product for given tasks, several comparison studies and web platforms, such as Medfloss.org, are available. However, due to the lack of a uniform terminology for health informatics, ambiguous or imprecise terms are used to describe the functionalities of LIFOSS. This makes comparisons of LIFOSS difficult and may lead to inappropriate software selection decisions. Using Linked Open Data (LOD) promises to address these challenges. Objective: We describe LIFOSS systematically with the help of the underlying Health Information Technology Ontology (HITO). We publish HITO and HITO-based software product descriptions using LOD to obtain the following benefits: (1) linking and reusing existing terminologies and (2) using Semantic Web tools for viewing and querying the LIFOSS data on the World Wide Web. Methods: HITO was incrementally developed and implemented. First, classes for the description of software products in health IT evaluation studies were identified. Second, requirements for describing LIFOSS were elicited by interviewing domain experts. Third, to describe domain-specific functionalities of software products, existing catalogues of features and enterprise functions were analyzed and integrated into the HITO knowledge base. As a proof of concept, HITO was used to describe 25 LIFOSS products. Results: HITO provides a defined set of classes and their relationships to describe LIFOSS in medicine and health care. With the help of linked or integrated catalogues for languages, programming languages, licenses, features, and enterprise functions, the functionalities of LIFOSS can be precisely described and compared. We publish HITO and the LIFOSS descriptions as LOD; they can be queried and viewed using different Semantic Web tools, such as Resource Description Framework (RDF) browsers, SPARQL Protocol and RDF Query Language (SPARQL) queries, and faceted searches. The advantages of providing HITO as LOD are demonstrated by practical examples. Conclusions: HITO is a building block to achieving unambiguous communication among health IT professionals and researchers. Providing LIFOSS product information as LOD enables barrier-free and easy access to data that are often hidden in user manuals of software products or are not available at all. Efforts to establish a unique terminology of medical and health informatics should be further supported and continued. UR - https://medinform.jmir.org/2023/1/e38861 UR - http://dx.doi.org/10.2196/38861 UR - http://www.ncbi.nlm.nih.gov/pubmed/36662569 ID - info:doi/10.2196/38861 ER - TY - JOUR AU - Jing, Xia AU - Min, Hua AU - Gong, Yang AU - Biondich, Paul AU - Robinson, David AU - Law, Timothy AU - Nohr, Christian AU - Faxvaag, Arild AU - Rennert, Lior AU - Hubig, Nina AU - Gimbel, Ronald PY - 2023/1/19 TI - Ontologies Applied in Clinical Decision Support System Rules: Systematic Review JO - JMIR Med Inform SP - e43053 VL - 11 KW - clinical decision support system rules KW - clinical decision support systems KW - interoperability KW - ontology KW - Semantic Web technology N2 - Background: Clinical decision support systems (CDSSs) are important for the quality and safety of health care delivery. Although CDSS rules guide CDSS behavior, they are not routinely shared and reused. Objective: Ontologies have the potential to promote the reuse of CDSS rules. Therefore, we systematically screened the literature to elaborate on the current status of ontologies applied in CDSS rules, such as rule management, which uses captured CDSS rule usage data and user feedback data to tailor CDSS services to be more accurate, and maintenance, which updates CDSS rules. Through this systematic literature review, we aim to identify the frontiers of ontologies used in CDSS rules. Methods: The literature search was focused on the intersection of ontologies; clinical decision support; and rules in PubMed, the Association for Computing Machinery (ACM) Digital Library, and the Nursing & Allied Health Database. Grounded theory and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 guidelines were followed. One author initiated the screening and literature review, while 2 authors validated the processes and results independently. The inclusion and exclusion criteria were developed and refined iteratively. Results: CDSSs were primarily used to manage chronic conditions, alerts for medication prescriptions, reminders for immunizations and preventive services, diagnoses, and treatment recommendations among 81 included publications. The CDSS rules were presented in Semantic Web Rule Language, Jess, or Jena formats. Despite the fact that ontologies have been used to provide medical knowledge, CDSS rules, and terminologies, they have not been used in CDSS rule management or to facilitate the reuse of CDSS rules. Conclusions: Ontologies have been used to organize and represent medical knowledge, controlled vocabularies, and the content of CDSS rules. So far, there has been little reuse of CDSS rules. More work is needed to improve the reusability and interoperability of CDSS rules. This review identified and described the ontologies that, despite their limitations, enable Semantic Web technologies and their applications in CDSS rules. UR - https://medinform.jmir.org/2023/1/e43053 UR - http://dx.doi.org/10.2196/43053 UR - http://www.ncbi.nlm.nih.gov/pubmed/36534739 ID - info:doi/10.2196/43053 ER - TY - JOUR AU - Lokala, Usha AU - Lamy, Francois AU - Daniulaityte, Raminta AU - Gaur, Manas AU - Gyrard, Amelie AU - Thirunarayan, Krishnaprasad AU - Kursuncu, Ugur AU - Sheth, Amit PY - 2022/12/23 TI - Drug Abuse Ontology to Harness Web-Based Data for Substance Use Epidemiology Research: Ontology Development Study JO - JMIR Public Health Surveill SP - e24938 VL - 8 IS - 12 KW - ontology KW - knowledge graph KW - semantic web KW - illicit drugs KW - cryptomarket KW - social media N2 - Background: Web-based resources and social media platforms play an increasingly important role in health-related knowledge and experience sharing. There is a growing interest in the use of these novel data sources for epidemiological surveillance of substance use behaviors and trends. Objective: The key aims were to describe the development and application of the drug abuse ontology (DAO) as a framework for analyzing web-based and social media data to inform public health and substance use research in the following areas: determining user knowledge, attitudes, and behaviors related to nonmedical use of buprenorphine and illicitly manufactured opioids through the analysis of web forum data Prescription Drug Abuse Online Surveillance; analyzing patterns and trends of cannabis product use in the context of evolving cannabis legalization policies in the United States through analysis of Twitter and web forum data (eDrugTrends); assessing trends in the availability of novel synthetic opioids through the analysis of cryptomarket data (eDarkTrends); and analyzing COVID-19 pandemic trends in social media data related to 13 states in the United States as per Mental Health America reports. Methods: The domain and scope of the DAO were defined using competency questions from popular ontology methodology (101 ontology development). The 101 method includes determining the domain and scope of ontology, reusing existing knowledge, enumerating important terms in ontology, defining the classes, their properties and creating instances of the classes. The quality of the ontology was evaluated using a set of tools and best practices recognized by the semantic web community and the artificial intelligence community that engage in natural language processing. Results: The current version of the DAO comprises 315 classes, 31 relationships, and 814 instances among the classes. The ontology is flexible and can easily accommodate new concepts. The integration of the ontology with machine learning algorithms dramatically decreased the false alarm rate by adding external knowledge to the machine learning process. The ontology is recurrently updated to capture evolving concepts in different contexts and applied to analyze data related to social media and dark web marketplaces. Conclusions: The DAO provides a powerful framework and a useful resource that can be expanded and adapted to a wide range of substance use and mental health domains to help advance big data analytics of web-based data for substance use epidemiology research. UR - https://publichealth.jmir.org/2022/12/e24938 UR - http://dx.doi.org/10.2196/24938 UR - http://www.ncbi.nlm.nih.gov/pubmed/36563032 ID - info:doi/10.2196/24938 ER - TY - JOUR AU - Binsfeld Gonçalves, Laurent AU - Nesic, Ivan AU - Obradovic, Marko AU - Stieltjes, Bram AU - Weikert, Thomas AU - Bremerich, Jens PY - 2022/12/21 TI - Natural Language Processing and Graph Theory: Making Sense of Imaging Records in a Novel Representation Frame JO - JMIR Med Inform SP - e40534 VL - 10 IS - 12 KW - radiology KW - deep learning KW - NLP KW - radiology reports KW - imaging record KW - temporal referrals KW - date extraction KW - graph theory KW - health care information system KW - resource planning. N2 - Background: A concise visualization framework of related reports would increase readability and improve patient management. To this end, temporal referrals to prior comparative exams are an essential connection to previous exams in written reports. Due to unstructured narrative texts' variable structure and content, their extraction is hampered by poor computer readability. Natural language processing (NLP) permits the extraction of structured information from unstructured texts automatically and can serve as an essential input for such a novel visualization framework. Objective: This study proposes and evaluates an NLP-based algorithm capable of extracting the temporal referrals in written radiology reports, applies it to all the radiology reports generated for 10 years, introduces a graphical representation of imaging reports, and investigates its benefits for clinical and research purposes. Methods: In this single-center, university hospital, retrospective study, we developed a convolutional neural network capable of extracting the date of referrals from imaging reports. The model's performance was assessed by calculating precision, recall, and F1-score using an independent test set of 149 reports. Next, the algorithm was applied to our department's radiology reports generated from 2011 to 2021. Finally, the reports and their metadata were represented in a modulable graph. Results: For extracting the date of referrals, the named-entity recognition (NER) model had a high precision of 0.93, a recall of 0.95, and an F1-score of 0.94. A total of 1,684,635 reports were included in the analysis. Temporal reference was mentioned in 53.3% (656,852/1,684,635), explicitly stated as not available in 21.0% (258,386/1,684,635), and omitted in 25.7% (317,059/1,684,635) of the reports. Imaging records can be visualized in a directed and modulable graph, in which the referring links represent the connecting arrows. Conclusions: Automatically extracting the date of referrals from unstructured radiology reports using deep learning NLP algorithms is feasible. Graphs refined the selection of distinct pathology pathways, facilitated the revelation of missing comparisons, and enabled the query of specific referring exam sequences. Further work is needed to evaluate its benefits in clinics, research, and resource planning. UR - https://medinform.jmir.org/2022/12/e40534 UR - http://dx.doi.org/10.2196/40534 UR - http://www.ncbi.nlm.nih.gov/pubmed/36542426 ID - info:doi/10.2196/40534 ER - TY - JOUR AU - Fan, ZhiYuan AU - Cui, LiYuan AU - Ye, Ying AU - Li, ShouCheng AU - Deng, Ning PY - 2022/12/19 TI - Telehealth System Based on the Ontology Design of a Diabetes Management Pathway Model in China: Development and Usability Study JO - JMIR Med Inform SP - e42664 VL - 10 IS - 12 KW - diabetes KW - chronic disease management KW - Chronic Disease Management Pathway KW - ontology KW - Semantic Web Rule Language rules KW - SWRL rules N2 - Background: Diabetes needs to be under control through management and intervention. Management of diabetes through mobile health is a practical approach; however, most diabetes mobile health management systems do not meet expectations, which may be because of the lack of standardized management processes in the systems and the lack of intervention implementation recommendations in the management knowledge base. Objective: In this study, we aimed to construct a diabetes management care pathway suitable for the actual situation in China to express the diabetes management care pathway using ontology and develop a diabetes closed-loop system based on the construction results of the diabetes management pathway and apply it practically. Methods: This study proposes a diabetes management care pathway model in which the management process of diabetes is divided into 9 management tasks, and the Diabetes Care Pathway Ontology (DCPO) is constructed to represent the knowledge contained in this pathway model. A telehealth system, which can support the comprehensive management of patients with diabetes while providing active intervention by physicians, was designed and developed based on the DCPO. A retrospective study was performed based on the data records extracted from the system to analyze the usability and treatment effects of the DCPO. Results: The diabetes management pathway ontology constructed in this study contains 119 newly added classes, 28 object properties, 58 data properties, 81 individuals, 426 axioms, and 192 Semantic Web Rule Language rules. The developed mobile medical system was applied to 272 patients with diabetes. Within 3 months, the average fasting blood glucose of the patients decreased by 1.34 mmol/L (P=.003), and the average 2-hour postprandial blood glucose decreased by 2.63 mmol/L (P=.003); the average systolic and diastolic blood pressures decreased by 11.84 mmHg (P=.02) and 8.8 mmHg (P=.02), respectively. In patients who received physician interventions owing to abnormal attention or low-compliance warnings, the average fasting blood glucose decreased by 2.45 mmol/L (P=.003), and the average 2-hour postprandial blood glucose decreased by 2.89 mmol/L (P=.003) in all patients with diabetes; the average systolic and diastolic blood pressure decreased by 20.06 mmHg (P=.02) and 17.37 mmHg (P=.02), respectively, in patients with both hypertension and diabetes during the 3-month management period. Conclusions: This study helps guide the timing and content of interactive interventions between physicians and patients and regulates physicians? medical service behavior. Different management plans are formulated for physicians and patients according to different characteristics to comprehensively manage various cardiovascular risk factors. The application of the DCPO in the diabetes management system can provide effective and adequate management support for patients with diabetes and those with both diabetes and hypertension. UR - https://medinform.jmir.org/2022/12/e42664 UR - http://dx.doi.org/10.2196/42664 UR - http://www.ncbi.nlm.nih.gov/pubmed/36534448 ID - info:doi/10.2196/42664 ER - TY - JOUR AU - Gérardin, Christel AU - Mageau, Arthur AU - Mékinian, Arsène AU - Tannier, Xavier AU - Carrat, Fabrice PY - 2022/12/19 TI - Construction of Cohorts of Similar Patients From Automatic Extraction of Medical Concepts: Phenotype Extraction Study JO - JMIR Med Inform SP - e42379 VL - 10 IS - 12 KW - natural language processing KW - similar patient cohort KW - phenotype KW - systemic disease KW - NLP KW - algorithm KW - automatic extraction KW - automated extraction KW - named entity KW - MeSH KW - medical subject heading KW - data extraction KW - text extraction N2 - Background: Reliable and interpretable automatic extraction of clinical phenotypes from large electronic medical record databases remains a challenge, especially in a language other than English. Objective: We aimed to provide an automated end-to-end extraction of cohorts of similar patients from electronic health records for systemic diseases. Methods: Our multistep algorithm includes a named-entity recognition step, a multilabel classification using medical subject headings ontology, and the computation of patient similarity. A selection of cohorts of similar patients on a priori annotated phenotypes was performed. Six phenotypes were selected for their clinical significance: P1, osteoporosis; P2, nephritis in systemic erythematosus lupus; P3, interstitial lung disease in systemic sclerosis; P4, lung infection; P5, obstetric antiphospholipid syndrome; and P6, Takayasu arteritis. We used a training set of 151 clinical notes and an independent validation set of 256 clinical notes, with annotated phenotypes, both extracted from the Assistance Publique-Hôpitaux de Paris data warehouse. We evaluated the precision of the 3 patients closest to the index patient for each phenotype with precision-at-3 and recall and average precision. Results: For P1-P4, the precision-at-3 ranged from 0.85 (95% CI 0.75-0.95) to 0.99 (95% CI 0.98-1), the recall ranged from 0.53 (95% CI 0.50-0.55) to 0.83 (95% CI 0.81-0.84), and the average precision ranged from 0.58 (95% CI 0.54-0.62) to 0.88 (95% CI 0.85-0.90). P5-P6 phenotypes could not be analyzed due to the limited number of phenotypes. Conclusions: Using a method close to clinical reasoning, we built a scalable and interpretable end-to-end algorithm for extracting cohorts of similar patients. UR - https://medinform.jmir.org/2022/12/e42379 UR - http://dx.doi.org/10.2196/42379 UR - http://www.ncbi.nlm.nih.gov/pubmed/36534446 ID - info:doi/10.2196/42379 ER - TY - JOUR AU - Kosowan, Leanne AU - Singer, Alexander AU - Zulkernine, Farhana AU - Zafari, Hasan AU - Nesca, Marcello AU - Muthumuni, Dhasni PY - 2022/12/13 TI - Pan-Canadian Electronic Medical Record Diagnostic and Unstructured Text Data for Capturing PTSD: Retrospective Observational Study JO - JMIR Med Inform SP - e41312 VL - 10 IS - 12 KW - electronic health records KW - EHR KW - natural language processing KW - NLP KW - medical informatics KW - primary health care KW - stress disorders, posttraumatic KW - posttraumatic stress disorder KW - PTSD N2 - Background: The availability of electronic medical record (EMR) free-text data for research varies. However, access to short diagnostic text fields is more widely available. Objective: This study assesses agreement between free-text and short diagnostic text data from primary care EMR for identification of posttraumatic stress disorder (PTSD). Methods: This retrospective cross-sectional study used EMR data from a pan-Canadian repository representing 1574 primary care providers at 265 clinics using 11 EMR vendors. Medical record review using free text and short diagnostic text fields of the EMR produced reference standards for PTSD. Agreement was assessed with sensitivity, specificity, positive predictive value, negative predictive value, and accuracy. Results: Our reference set contained 327 patients with free text and short diagnostic text. Among these patients, agreement between free text and short diagnostic text had an accuracy of 93.6% (CI 90.4%-96.0%). In a single Canadian province, case definitions 1 and 4 had a sensitivity of 82.6% (CI 74.4%-89.0%) and specificity of 99.5% (CI 97.4%-100%). However, when the reference set was expanded to a pan-Canada reference (n=12,104 patients), case definition 4 had the strongest agreement (sensitivity: 91.1%, CI 90.1%-91.9%; specificity: 99.1%, CI 98.9%-99.3%). Conclusions: Inclusion of free-text encounter notes during medical record review did not lead to improved capture of PTSD cases, nor did it lead to significant changes in case definition agreement. Within this pan-Canadian database, jurisdictional differences in diagnostic codes and EMR structure suggested the need to supplement diagnostic codes with natural language processing to capture PTSD. When unavailable, short diagnostic text can supplement free-text data for reference set creation and case validation. Application of the PTSD case definition can inform PTSD prevalence and characteristics. UR - https://medinform.jmir.org/2022/12/e41312 UR - http://dx.doi.org/10.2196/41312 UR - http://www.ncbi.nlm.nih.gov/pubmed/36512389 ID - info:doi/10.2196/41312 ER - TY - JOUR AU - Han, Feng AU - Zhang, ZiHeng AU - Zhang, Hongjian AU - Nakaya, Jun AU - Kudo, Kohsuke AU - Ogasawara, Katsuhiko PY - 2022/11/18 TI - Extraction and Quantification of Words Representing Degrees of Diseases: Combining the Fuzzy C-Means Method and Gaussian Membership JO - JMIR Form Res SP - e38677 VL - 6 IS - 11 KW - medical text KW - fuzzy c-means KW - cluster KW - algorithm KW - machine learning KW - word quantification KW - fuzzification KW - Gauss KW - radiology KW - medical report KW - documentation KW - text mining KW - data mining KW - extraction KW - unstructured KW - free text KW - quantification KW - fuzzy KW - diagnosis KW - diagnostic KW - EHR KW - support system N2 - Background: Due to the development of medical data, a large amount of clinical data has been generated. These unstructured data contain substantial information. Extracting useful knowledge from this data and making scientific decisions for diagnosing and treating diseases have become increasingly necessary. Unstructured data, such as in the Marketplace for Medical Information in Intensive Care III (MIMIC-III) data set, contain several ambiguous words that demonstrate the subjectivity of doctors, such as descriptions of patient symptoms. These data could be used to further improve the accuracy of medical diagnostic system assessments. To the best of our knowledge, there is currently no method for extracting subjective words that express the extent of these symptoms (hereinafter, ?degree words?). Objective: Therefore, we propose using the fuzzy c-means (FCM) method and Gaussian membership to quantify the degree words in the clinical medical data set MIMIC-III. Methods: First, we preprocessed the 381,091 radiology reports collected in MIMIC-III, and then we used the FCM method to extract degree words from unstructured text. Thereafter, we used the Gaussian membership method to quantify the extracted degree words, which transform the fuzzy words extracted from the medical text into computer-recognizable numbers. Results: The results showed that the digitization of ambiguous words in medical texts is feasible. The words representing each degree of each disease had a range of corresponding values. Examples of membership medians were 2.971 (atelectasis), 3.121 (pneumonia), 2.899 (pneumothorax), 3.051 (pulmonary edema), and 2.435 (pulmonary embolus). Additionally, all extracted words contained the same subjective words (low, high, etc), which allows for an objective evaluation method. Furthermore, we will verify the specific impact of the quantification results of ambiguous words such as symptom words and degree words on the use of medical texts in subsequent studies. These same ambiguous words may be used as a new set of feature values to represent the disorders. Conclusions: This study proposes an innovative method for handling subjective words. We used the FCM method to extract the subjective degree words in the English-interpreted report of the MIMIC-III and then used the Gaussian functions to quantify the subjective degree words. In this method, words containing subjectivity in unstructured texts can be automatically processed and transformed into numerical ranges by digital processing. It was concluded that the digitization of ambiguous words in medical texts is feasible. UR - https://formative.jmir.org/2022/11/e38677 UR - http://dx.doi.org/10.2196/38677 UR - http://www.ncbi.nlm.nih.gov/pubmed/36399376 ID - info:doi/10.2196/38677 ER - TY - JOUR AU - Chen, Pei-Fu AU - He, Tai-Liang AU - Lin, Sheng-Che AU - Chu, Yuan-Chia AU - Kuo, Chen-Tsung AU - Lai, Feipei AU - Wang, Ssu-Ming AU - Zhu, Wan-Xuan AU - Chen, Kuan-Chih AU - Kuo, Lu-Cheng AU - Hung, Fang-Ming AU - Lin, Yu-Cheng AU - Tsai, I-Chang AU - Chiu, Chi-Hao AU - Chang, Shu-Chih AU - Yang, Chi-Yu PY - 2022/11/10 TI - Training a Deep Contextualized Language Model for International Classification of Diseases, 10th Revision Classification via Federated Learning: Model Development and Validation Study JO - JMIR Med Inform SP - e41342 VL - 10 IS - 11 KW - federated learning KW - International Classification of Diseases KW - machine learning KW - natural language processing KW - multilabel text classification N2 - Background: The automatic coding of clinical text documents by using the International Classification of Diseases, 10th Revision (ICD-10) can be performed for statistical analyses and reimbursements. With the development of natural language processing models, new transformer architectures with attention mechanisms have outperformed previous models. Although multicenter training may increase a model?s performance and external validity, the privacy of clinical documents should be protected. We used federated learning to train a model with multicenter data, without sharing data per se. Objective: This study aims to train a classification model via federated learning for ICD-10 multilabel classification. Methods: Text data from discharge notes in electronic medical records were collected from the following three medical centers: Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital. After comparing the performance of different variants of bidirectional encoder representations from transformers (BERT), PubMedBERT was chosen for the word embeddings. With regard to preprocessing, the nonalphanumeric characters were retained because the model?s performance decreased after the removal of these characters. To explain the outputs of our model, we added a label attention mechanism to the model architecture. The model was trained with data from each of the three hospitals separately and via federated learning. The models trained via federated learning and the models trained with local data were compared on a testing set that was composed of data from the three hospitals. The micro F1 score was used to evaluate model performance across all 3 centers. Results: The F1 scores of PubMedBERT, RoBERTa (Robustly Optimized BERT Pretraining Approach), ClinicalBERT, and BioBERT (BERT for Biomedical Text Mining) were 0.735, 0.692, 0.711, and 0.721, respectively. The F1 score of the model that retained nonalphanumeric characters was 0.8120, whereas the F1 score after removing these characters was 0.7875?a decrease of 0.0245 (3.11%). The F1 scores on the testing set were 0.6142, 0.4472, 0.5353, and 0.2522 for the federated learning, Far Eastern Memorial Hospital, National Taiwan University Hospital, and Taipei Veterans General Hospital models, respectively. The explainable predictions were displayed with highlighted input words via the label attention architecture. Conclusions: Federated learning was used to train the ICD-10 classification model on multicenter clinical text while protecting data privacy. The model?s performance was better than that of models that were trained locally. UR - https://medinform.jmir.org/2022/11/e41342 UR - http://dx.doi.org/10.2196/41342 UR - http://www.ncbi.nlm.nih.gov/pubmed/36355417 ID - info:doi/10.2196/41342 ER - TY - JOUR AU - Guardiolle, Vianney AU - Bazoge, Adrien AU - Morin, Emmanuel AU - Daille, Béatrice AU - Toublant, Delphine AU - Bouzillé, Guillaume AU - Merel, Youenn AU - Pierre-Jean, Morgane AU - Filiot, Alexandre AU - Cuggia, Marc AU - Wargny, Matthieu AU - Lamer, Antoine AU - Gourraud, Pierre-Antoine PY - 2022/11/1 TI - Linking Biomedical Data Warehouse Records With the National Mortality Database in France: Large-scale Matching Algorithm JO - JMIR Med Inform SP - e36711 VL - 10 IS - 11 KW - data warehousing KW - clinical data warehouse KW - medical informatics applications KW - medical record linkage KW - French National Mortality Database KW - data reuse KW - open data, R KW - clinical informatics N2 - Background: Often missing from or uncertain in a biomedical data warehouse (BDW), vital status after discharge is central to the value of a BDW in medical research. The French National Mortality Database (FNMD) offers open-source nominative records of every death. Matching large-scale BDWs records with the FNMD combines multiple challenges: absence of unique common identifiers between the 2 databases, names changing over life, clerical errors, and the exponential growth of the number of comparisons to compute. Objective: We aimed to develop a new algorithm for matching BDW records to the FNMD and evaluated its performance. Methods: We developed a deterministic algorithm based on advanced data cleaning and knowledge of the naming system and the Damerau-Levenshtein distance (DLD). The algorithm?s performance was independently assessed using BDW data of 3 university hospitals: Lille, Nantes, and Rennes. Specificity was evaluated with living patients on January 1, 2016 (ie, patients with at least 1 hospital encounter before and after this date). Sensitivity was evaluated with patients recorded as deceased between January 1, 2001, and December 31, 2020. The DLD-based algorithm was compared to a direct matching algorithm with minimal data cleaning as a reference. Results: All centers combined, sensitivity was 11% higher for the DLD-based algorithm (93.3%, 95% CI 92.8-93.9) than for the direct algorithm (82.7%, 95% CI 81.8-83.6; P<.001). Sensitivity was superior for men at 2 centers (Nantes: 87%, 95% CI 85.1-89 vs 83.6%, 95% CI 81.4-85.8; P=.006; Rennes: 98.6%, 95% CI 98.1-99.2 vs 96%, 95% CI 94.9-97.1; P<.001) and for patients born in France at all centers (Nantes: 85.8%, 95% CI 84.3-87.3 vs 74.9%, 95% CI 72.8-77.0; P<.001). The DLD-based algorithm revealed significant differences in sensitivity among centers (Nantes, 85.3% vs Lille and Rennes, 97.3%, P<.001). Specificity was >98% in all subgroups. Our algorithm matched tens of millions of death records from BDWs, with parallel computing capabilities and low RAM requirements. We used the Inseehop open-source R script for this measurement. Conclusions: Overall, sensitivity/recall was 11% higher using the DLD-based algorithm than that using the direct algorithm. This shows the importance of advanced data cleaning and knowledge of a naming system through DLD use. Statistically significant differences in sensitivity between groups could be found and must be considered when performing an analysis to avoid differential biases. Our algorithm, originally conceived for linking a BDW with the FNMD, can be used to match any large-scale databases. While matching operations using names are considered sensitive computational operations, the Inseehop package released here is easy to run on premises, thereby facilitating compliance with cybersecurity local framework. The use of an advanced deterministic matching algorithm such as the DLD-based algorithm is an insightful example of combining open-source external data to improve the usage value of BDWs. UR - https://medinform.jmir.org/2022/11/e36711 UR - http://dx.doi.org/10.2196/36711 UR - http://www.ncbi.nlm.nih.gov/pubmed/36318244 ID - info:doi/10.2196/36711 ER - TY - JOUR AU - Park, H. Eunsoo AU - Watson, I. Hannah AU - Mehendale, V. Felicity AU - O'Neil, Q. Alison AU - PY - 2022/10/26 TI - Evaluating the Impact on Clinical Task Efficiency of a Natural Language Processing Algorithm for Searching Medical Documents: Prospective Crossover Study JO - JMIR Med Inform SP - e39616 VL - 10 IS - 10 KW - clinical decision support KW - electronic health records KW - natural language processing KW - semantic search KW - clinical informatics N2 - Background: Information retrieval (IR) from the free text within electronic health records (EHRs) is time consuming and complex. We hypothesize that natural language processing (NLP)?enhanced search functionality for EHRs can make clinical workflows more efficient and reduce cognitive load for clinicians. Objective: This study aimed to evaluate the efficacy of 3 levels of search functionality (no search, string search, and NLP-enhanced search) in supporting IR for clinical users from the free text of EHR documents in a simulated clinical environment. Methods: A clinical environment was simulated by uploading 3 sets of patient notes into an EHR research software application and presenting these alongside 3 corresponding IR tasks. Tasks contained a mixture of multiple-choice and free-text questions. A prospective crossover study design was used, for which 3 groups of evaluators were recruited, which comprised doctors (n=19) and medical students (n=16). Evaluators performed the 3 tasks using each of the search functionalities in an order in accordance with their randomly assigned group. The speed and accuracy of task completion were measured and analyzed, and user perceptions of NLP-enhanced search were reviewed in a feedback survey. Results: NLP-enhanced search facilitated more accurate task completion than both string search (5.14%; P=.02) and no search (5.13%; P=.08). NLP-enhanced search and string search facilitated similar task speeds, both showing an increase in speed compared to the no search function, by 11.5% (P=.008) and 16.0% (P=.007) respectively. Overall, 93% of evaluators agreed that NLP-enhanced search would make clinical workflows more efficient than string search, with qualitative feedback reporting that NLP-enhanced search reduced cognitive load. Conclusions: To the best of our knowledge, this study is the largest evaluation to date of different search functionalities for supporting target clinical users in realistic clinical workflows, with a 3-way prospective crossover study design. NLP-enhanced search improved both accuracy and speed of clinical EHR IR tasks compared to browsing clinical notes without search. NLP-enhanced search improved accuracy and reduced the number of searches required for clinical EHR IR tasks compared to direct search term matching. UR - https://medinform.jmir.org/2022/10/e39616 UR - http://dx.doi.org/10.2196/39616 UR - http://www.ncbi.nlm.nih.gov/pubmed/36287591 ID - info:doi/10.2196/39616 ER - TY - JOUR AU - Oates, John AU - Shafiabady, Niusha AU - Ambagtsheer, Rachel AU - Beilby, Justin AU - Seiboth, Chris AU - Dent, Elsa PY - 2022/10/7 TI - Evolving Hybrid Partial Genetic Algorithm Classification Model for Cost-effective Frailty Screening: Investigative Study JO - JMIR Aging SP - e38464 VL - 5 IS - 4 KW - machine learning KW - frailty screening KW - partial genetic algorithms KW - SVM KW - KNN KW - decision trees KW - frailty KW - algorithm KW - cost KW - model KW - index KW - database KW - ai KW - ageing KW - adults KW - older people KW - screening KW - tool N2 - Background: A commonly used method for measuring frailty is the accumulation of deficits expressed as a frailty index (FI). FIs can be readily adapted to many databases, as the parameters to use are not prescribed but rather reflect a subset of extracted features (variables). Unfortunately, the structure of many databases does not permit the direct extraction of a suitable subset, requiring additional effort to determine and verify the value of features for each record and thus significantly increasing cost. Objective: Our objective is to describe how an artificial intelligence (AI) optimization technique called partial genetic algorithms can be used to refine the subset of features used to calculate an FI and favor features that have the least cost of acquisition. Methods: This is a secondary analysis of a residential care database compiled from 10 facilities in Queensland, Australia. The database is comprised of routinely collected administrative data and unstructured patient notes for 592 residents aged 75 years and over. The primary study derived an electronic frailty index (eFI) calculated from 36 suitable features. We then structurally modified a genetic algorithm to find an optimal predictor of the calculated eFI (0.21 threshold) from 2 sets of features. Partial genetic algorithms were used to optimize 4 underlying classification models: logistic regression, decision trees, random forest, and support vector machines. Results: Among the underlying models, logistic regression was found to produce the best models in almost all scenarios and feature set sizes. The best models were built using all the low-cost features and as few as 10 high-cost features, and they performed well enough (sensitivity 89%, specificity 87%) to be considered candidates for a low-cost frailty screening test. Conclusions: In this study, a systematic approach for selecting an optimal set of features with a low cost of acquisition and performance comparable to the eFI for detecting frailty was demonstrated on an aged care database. Partial genetic algorithms have proven useful in offering a trade-off between cost and accuracy to systematically identify frailty. UR - https://aging.jmir.org/2022/4/e38464 UR - http://dx.doi.org/10.2196/38464 UR - http://www.ncbi.nlm.nih.gov/pubmed/36206042 ID - info:doi/10.2196/38464 ER - TY - JOUR AU - Mahmoudi, Elham AU - Wu, Wenbo AU - Najarian, Cyrus AU - Aikens, James AU - Bynum, Julie AU - Vydiswaran, Vinod V. G. PY - 2022/9/22 TI - Identifying Caregiver Availability Using Medical Notes With Rule-Based Natural Language Processing: Retrospective Cohort Study JO - JMIR Aging SP - e40241 VL - 5 IS - 3 KW - natural language processing KW - caregiver KW - medical notes KW - Alzheimer KW - dementia KW - pragmatic KW - aging KW - care planning KW - health care KW - elderly care KW - elderly population KW - algorithm N2 - Background: Identifying caregiver availability, particularly for patients with dementia or those with a disability, is critical to informing the appropriate care planning by the health systems, hospitals, and providers. This information is not readily available, and there is a paucity of pragmatic approaches to automatically identifying caregiver availability and type. Objective: Our main objective was to use medical notes to assess caregiver availability and type for hospitalized patients with dementia. Our second objective was to identify whether the patient lived at home or resided at an institution. Methods: In this retrospective cohort study, we used 2016-2019 telephone-encounter medical notes from a single institution to develop a rule-based natural language processing (NLP) algorithm to identify the patient?s caregiver availability and place of residence. Using note-level data, we compared the results of the NLP algorithm with human-conducted chart abstraction for both training (749/976, 77%) and test sets (227/976, 23%) for a total of 223 adults aged 65 years and older diagnosed with dementia. Our outcomes included determining whether the patients (1) reside at home or in an institution, (2) have a formal caregiver, and (3) have an informal caregiver. Results: Test set results indicated that our NLP algorithm had high level of accuracy and reliability for identifying whether patients had an informal caregiver (F1=0.94, accuracy=0.95, sensitivity=0.97, and specificity=0.93), but was relatively less able to identify whether the patient lived at an institution (F1=0.64, accuracy=0.90, sensitivity=0.51, and specificity=0.98). The most common explanations for NLP misclassifications across all categories were (1) incomplete or misspelled facility names; (2) past, uncertain, or undecided status; (3) uncommon abbreviations; and (4) irregular use of templates. Conclusions: This innovative work was the first to use medical notes to pragmatically determine caregiver availability. Our NLP algorithm identified whether hospitalized patients with dementia have a formal or informal caregiver and, to a lesser extent, whether they lived at home or in an institutional setting. There is merit in using NLP to identify caregivers. This study serves as a proof of concept. Future work can use other approaches and further identify caregivers and the extent of their availability. UR - https://aging.jmir.org/2022/3/e40241 UR - http://dx.doi.org/10.2196/40241 UR - http://www.ncbi.nlm.nih.gov/pubmed/35998328 ID - info:doi/10.2196/40241 ER - TY - JOUR AU - Ferrell, J. Brian AU - Raskin, E. Sarah AU - Zimmerman, B. Emily AU - Timberline, H. David AU - McInnes, T. Bridget AU - Krist, H. Alex PY - 2022/9/6 TI - Attention-Based Models for Classifying Small Data Sets Using Community-Engaged Research Protocols: Classification System Development and Validation Pilot Study JO - JMIR Form Res SP - e32460 VL - 6 IS - 9 KW - data augmentation KW - BERT KW - transformer-based models KW - text classification KW - community engagement KW - prototype KW - IRB research KW - community-engaged research KW - participatory research KW - deep learning N2 - Background: Community-engaged research (CEnR) is a research approach in which scholars partner with community organizations or individuals with whom they share an interest in the study topic, typically with the goal of supporting that community?s well-being. CEnR is well-established in numerous disciplines including the clinical and social sciences. However, universities experience challenges reporting comprehensive CEnR metrics, limiting the development of appropriate CEnR infrastructure and the advancement of relationships with communities, funders, and stakeholders. Objective: We propose a novel approach to identifying and categorizing community-engaged studies by applying attention-based deep learning models to human participants protocols that have been submitted to the university?s institutional review board (IRB). Methods: We manually classified a sample of 280 protocols submitted to the IRB using a 3- and 6-level CEnR heuristic. We then trained an attention-based bidirectional long short-term memory unit (Bi-LSTM) on the classified protocols and compared it to transformer models such as Bidirectional Encoder Representations From Transformers (BERT), Bio + Clinical BERT, and Cross-lingual Language Model?Robustly Optimized BERT Pre-training Approach (XLM-RoBERTa). We applied the best-performing models to the full sample of unlabeled IRB protocols submitted in the years 2013-2019 (n>6000). Results: Although transfer learning is superior, receiving a 0.9952 evaluation F1 score for all transformer models implemented compared to the attention-based Bi-LSTM (between 48%-80%), there were key issues with overfitting. This finding is consistent across several methodological adjustments: an augmented data set with and without cross-validation, an unaugmented data set with and without cross-validation, a 6-class CEnR spectrum, and a 3-class one. Conclusions: Transfer learning is a more viable method than the attention-based bidirectional-LSTM for differentiating small data sets characterized by the idiosyncrasies and variability of CEnR descriptions used by principal investigators in research protocols. Despite these issues involving overfitting, BERT and the other transformer models remarkably showed an understanding of our data unlike the attention-based Bi-LSTM model, promising a more realistic path toward solving this real-world application. UR - https://formative.jmir.org/2022/9/e32460 UR - http://dx.doi.org/10.2196/32460 UR - http://www.ncbi.nlm.nih.gov/pubmed/36066925 ID - info:doi/10.2196/32460 ER - TY - JOUR AU - Kiser, C. Amber AU - Eilbeck, Karen AU - Ferraro, P. Jeffrey AU - Skarda, E. David AU - Samore, H. Matthew AU - Bucher, Brian PY - 2022/8/30 TI - Standard Vocabularies to Improve Machine Learning Model Transferability With Electronic Health Record Data: Retrospective Cohort Study Using Health Care?Associated Infection JO - JMIR Med Inform SP - e39057 VL - 10 IS - 8 KW - standard vocabularies KW - machine learning KW - electronic health records KW - model transferability KW - data heterogeneity N2 - Background: With the widespread adoption of electronic healthcare records (EHRs) by US hospitals, there is an opportunity to leverage this data for the development of predictive algorithms to improve clinical care. A key barrier in model development and implementation includes the external validation of model discrimination, which is rare and often results in worse performance. One reason why machine learning models are not externally generalizable is data heterogeneity. A potential solution to address the substantial data heterogeneity between health care systems is to use standard vocabularies to map EHR data elements. The advantage of these vocabularies is a hierarchical relationship between elements, which allows the aggregation of specific clinical features to more general grouped concepts. Objective: This study aimed to evaluate grouping EHR data using standard vocabularies to improve the transferability of machine learning models for the detection of postoperative health care?associated infections across institutions with different EHR systems. Methods: Patients who underwent surgery from the University of Utah Health and Intermountain Healthcare from July 2014 to August 2017 with complete follow-up data were included. The primary outcome was a health care?associated infection within 30 days of the procedure. EHR data from 0-30 days after the operation were mapped to standard vocabularies and grouped using the hierarchical relationships of the vocabularies. Model performance was measured using the area under the receiver operating characteristic curve (AUC) and F1-score in internal and external validations. To evaluate model transferability, a difference-in-difference metric was defined as the difference in performance drop between internal and external validations for the baseline and grouped models. Results: A total of 5775 patients from the University of Utah and 15,434 patients from Intermountain Healthcare were included. The prevalence of selected outcomes was from 4.9% (761/15,434) to 5% (291/5775) for surgical site infections, from 0.8% (44/5775) to 1.1% (171/15,434) for pneumonia, from 2.6% (400/15,434) to 3% (175/5775) for sepsis, and from 0.8% (125/15,434) to 0.9% (50/5775) for urinary tract infections. In all outcomes, the grouping of data using standard vocabularies resulted in a reduced drop in AUC and F1-score in external validation compared to baseline features (all P<.001, except urinary tract infection AUC: P=.002). The difference-in-difference metrics ranged from 0.005 to 0.248 for AUC and from 0.075 to 0.216 for F1-score. Conclusions: We demonstrated that grouping machine learning model features based on standard vocabularies improved model transferability between data sets across 2 institutions. Improving model transferability using standard vocabularies has the potential to improve the generalization of clinical prediction models across the health care system. UR - https://medinform.jmir.org/2022/8/e39057 UR - http://dx.doi.org/10.2196/39057 UR - http://www.ncbi.nlm.nih.gov/pubmed/36040784 ID - info:doi/10.2196/39057 ER - TY - JOUR AU - Delanerolle, Gayathri AU - Williams, Robert AU - Stipancic, Ana AU - Byford, Rachel AU - Forbes, Anna AU - Tsang, M. Ruby S. AU - Anand, N. Sneha AU - Bradley, Declan AU - Murphy, Siobhán AU - Akbari, Ashley AU - Bedston, Stuart AU - Lyons, A. Ronan AU - Owen, Rhiannon AU - Torabi, Fatemeh AU - Beggs, Jillian AU - Chuter, Antony AU - Balharry, Dominique AU - Joy, Mark AU - Sheikh, Aziz AU - Hobbs, Richard F. D. AU - de Lusignan, Simon PY - 2022/8/22 TI - Methodological Issues in Using a Common Data Model of COVID-19 Vaccine Uptake and Important Adverse Events of Interest: Feasibility Study of Data and Connectivity COVID-19 Vaccines Pharmacovigilance in the United Kingdom JO - JMIR Form Res SP - e37821 VL - 6 IS - 8 KW - Systematized Nomenclature of Medicine KW - COVID-19 vaccines KW - COVID-19 KW - sinus thrombosis KW - anaphylaxis KW - pharmacovigilance KW - vaccine uptake KW - medical outcome KW - clinical coding system KW - health database KW - health information KW - clinical outcome KW - vaccine effect KW - data model N2 - Background: The Data and Connectivity COVID-19 Vaccines Pharmacovigilance (DaC-VaP) UK-wide collaboration was created to monitor vaccine uptake and effectiveness and provide pharmacovigilance using routine clinical and administrative data. To monitor these, pooled analyses may be needed. However, variation in terminologies present a barrier as England uses the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), while the rest of the United Kingdom uses the Read v2 terminology in primary care. The availability of data sources is not uniform across the United Kingdom. Objective: This study aims to use the concept mappings in the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to identify common concepts recorded and to report these in a repeated cross-sectional study. We planned to do this for vaccine coverage and 2 adverse events of interest (AEIs), cerebral venous sinus thrombosis (CVST) and anaphylaxis. We identified concept mappings to SNOMED CT, Read v2, the World Health Organization?s International Classification of Disease Tenth Revision (ICD-10) terminology, and the UK Dictionary of Medicines and Devices (dm+d). Methods: Exposures and outcomes of interest to DaC-VaP for pharmacovigilance studies were selected. Mappings of these variables to different terminologies used across the United Kingdom?s devolved nations? health services were identified from the Observational Health Data Sciences and Informatics (OHDSI) Automated Terminology Harmonization, Extraction, and Normalization for Analytics (ATHENA) online browser. Lead analysts from each nation then confirmed or added to the mappings identified. These mappings were then used to report AEIs in a common format. We reported rates for windows of 0-2 and 3-28 days postvaccine every 28 days. Results: We listed the mappings between Read v2, SNOMED CT, ICD-10, and dm+d. For vaccine exposure, we found clear mapping from OMOP to our clinical terminologies, though dm+d had codes not listed by OMOP at the time of searching. We found a list of CVST and anaphylaxis codes. For CVST, we had to use a broader cerebral venous thrombosis conceptual approach to include Read v2. We identified 56 SNOMED CT codes, of which we selected 47 (84%), and 15 Read v2 codes. For anaphylaxis, our refined search identified 60 SNOMED CT codes and 9 Read v2 codes, of which we selected 10 (17%) and 4 (44%), respectively, to include in our repeated cross-sectional studies. Conclusions: This approach enables the use of mappings to different terminologies within the OMOP CDM without the need to catalogue an entire database. However, Read v2 has less granular concepts than some terminologies, such as SNOMED CT. Additionally, the OMOP CDM cannot compensate for limitations in the clinical coding system. Neither Read v2 nor ICD-10 is sufficiently granular to enable CVST to be specifically flagged. Hence, any pooled analysis will have to be at the less specific level of cerebrovascular venous thrombosis. Overall, the mappings within this CDM are useful, and our method could be used for rapid collaborations where there are only a limited number of concepts to pool. UR - https://formative.jmir.org/2022/8/e37821 UR - http://dx.doi.org/10.2196/37821 UR - http://www.ncbi.nlm.nih.gov/pubmed/35786634 ID - info:doi/10.2196/37821 ER - TY - JOUR AU - Kaur, Manpreet AU - Costello, Jeremy AU - Willis, Elyse AU - Kelm, Karen AU - Reformat, Z. Marek AU - Bolduc, V. Francois PY - 2022/8/5 TI - Deciphering the Diversity of Mental Models in Neurodevelopmental Disorders: Knowledge Graph Representation of Public Data Using Natural Language Processing JO - J Med Internet Res SP - e39888 VL - 24 IS - 8 KW - concept map KW - neurodevelopmental disorder KW - knowledge graph KW - text analysis KW - semantic relatedness KW - PubMed KW - forums KW - mental model N2 - Background: Understanding how individuals think about a topic, known as the mental model, can significantly improve communication, especially in the medical domain where emotions and implications are high. Neurodevelopmental disorders (NDDs) represent a group of diagnoses, affecting up to 18% of the global population, involving differences in the development of cognitive or social functions. In this study, we focus on 2 NDDs, attention deficit hyperactivity disorder (ADHD) and autism spectrum disorder (ASD), which involve multiple symptoms and interventions requiring interactions between 2 important stakeholders: parents and health professionals. There is a gap in our understanding of differences between mental models for each stakeholder, making communication between stakeholders more difficult than it could be. Objective: We aim to build knowledge graphs (KGs) from web-based information relevant to each stakeholder as proxies of mental models. These KGs will accelerate the identification of shared and divergent concerns between stakeholders. The developed KGs can help improve knowledge mobilization, communication, and care for individuals with ADHD and ASD. Methods: We created 2 data sets by collecting the posts from web-based forums and PubMed abstracts related to ADHD and ASD. We utilized the Unified Medical Language System (UMLS) to detect biomedical concepts and applied Positive Pointwise Mutual Information followed by truncated Singular Value Decomposition to obtain corpus-based concept embeddings for each data set. Each data set is represented as a KG using a property graph model. Semantic relatedness between concepts is calculated to rank the relation strength of concepts and stored in the KG as relation weights. UMLS disorder-relevant semantic types are used to provide additional categorical information about each concept?s domain. Results: The developed KGs contain concepts from both data sets, with node sizes representing the co-occurrence frequency of concepts and edge sizes representing relevance between concepts. ADHD- and ASD-related concepts from different semantic types shows diverse areas of concerns and complex needs of the conditions. KG identifies converging and diverging concepts between health professionals literature (PubMed) and parental concerns (web-based forums), which may correspond to the differences between mental models for each stakeholder. Conclusions: We show for the first time that generating KGs from web-based data can capture the complex needs of families dealing with ADHD or ASD. Moreover, we showed points of convergence between families and health professionals? KGs. Natural language processing?based KG provides access to a large sample size, which is often a limiting factor for traditional in-person mental model mapping. Our work offers a high throughput access to mental model maps, which could be used for further in-person validation, knowledge mobilization projects, and basis for communication about potential blind spots from stakeholders in interactions about NDDs. Future research will be needed to identify how concepts could interact together differently for each stakeholder. UR - https://www.jmir.org/2022/8/e39888 UR - http://dx.doi.org/10.2196/39888 UR - http://www.ncbi.nlm.nih.gov/pubmed/35930346 ID - info:doi/10.2196/39888 ER - TY - JOUR AU - Tang, Wentai AU - Wang, Jian AU - Lin, Hongfei AU - Zhao, Di AU - Xu, Bo AU - Zhang, Yijia AU - Yang, Zhihao PY - 2022/8/2 TI - A Syntactic Information?Based Classification Model for Medical Literature: Algorithm Development and Validation Study JO - JMIR Med Inform SP - e37817 VL - 10 IS - 8 KW - medical relation extraction KW - syntactic features KW - pruning method KW - neural networks KW - medical literature KW - medical text KW - extraction KW - syntactic KW - classification KW - interaction KW - text KW - literature KW - semantic N2 - Background: The ever-increasing volume of medical literature necessitates the classification of medical literature. Medical relation extraction is a typical method of classifying a large volume of medical literature. With the development of arithmetic power, medical relation extraction models have evolved from rule-based models to neural network models. The single neural network model discards the shallow syntactic information while discarding the traditional rules. Therefore, we propose a syntactic information?based classification model that complements and equalizes syntactic information to enhance the model. Objective: We aim to complete a syntactic information?based relation extraction model for more efficient medical literature classification. Methods: We devised 2 methods for enhancing syntactic information in the model. First, we introduced shallow syntactic information into the convolutional neural network to enhance nonlocal syntactic interactions. Second, we devise a cross-domain pruning method to equalize local and nonlocal syntactic interactions. Results: We experimented with 3 data sets related to the classification of medical literature. The F1 values were 65.5% and 91.5% on the BioCreative ViCPR (CPR) and Phenotype-Gene Relationship data sets, respectively, and the accuracy was 88.7% on the PubMed data set. Our model outperforms the current state-of-the-art baseline model in the experiments. Conclusions: Our model based on syntactic information effectively enhances medical relation extraction. Furthermore, the results of the experiments show that shallow syntactic information helps obtain nonlocal interaction in sentences and effectively reinforces syntactic features. It also provides new ideas for future research directions. UR - https://medinform.jmir.org/2022/8/e37817 UR - http://dx.doi.org/10.2196/37817 UR - http://www.ncbi.nlm.nih.gov/pubmed/35917162 ID - info:doi/10.2196/37817 ER - TY - JOUR AU - Román-Villarán, Esther AU - Alvarez-Romero, Celia AU - Martínez-García, Alicia AU - Escobar-Rodríguez, Antonio German AU - García-Lozano, José María AU - Barón-Franco, Bosco AU - Moreno-Gaviño, Lourdes AU - Moreno-Conde, Jesús AU - Rivas-González, Antonio José AU - Parra-Calderón, Luis Carlos PY - 2022/8/2 TI - A Personalized Ontology-Based Decision Support System for Complex Chronic Patients: Retrospective Observational Study JO - JMIR Form Res SP - e27990 VL - 6 IS - 8 KW - adherence KW - ontology KW - clinical decision support system KW - CDSS KW - complex chronic patients KW - functional validation KW - multimorbidity KW - polypharmacy KW - atrial fibrillation KW - anticoagulants N2 - Background: Due to an increase in life expectancy, the prevalence of chronic diseases is also on the rise. Clinical practice guidelines (CPGs) provide recommendations for suitable interventions regarding different chronic diseases, but a deficiency in the implementation of these CPGs has been identified. The PITeS-TiiSS (Telemedicine and eHealth Innovation Platform: Information Communications Technology for Research and Information Challenges in Health Services) tool, a personalized ontology-based clinical decision support system (CDSS), aims to reduce variability, prevent errors, and consider interactions between different CPG recommendations, among other benefits. Objective: The aim of this study is to design, develop, and validate an ontology-based CDSS that provides personalized recommendations related to drug prescription. The target population is older adult patients with chronic diseases and polypharmacy, and the goal is to reduce complications related to these types of conditions while offering integrated care. Methods: A study scenario about atrial fibrillation and treatment with anticoagulants was selected to validate the tool. After this, a series of knowledge sources were identified, including CPGs, PROFUND index, LESS/CHRON criteria, and STOPP/START criteria, to extract the information. Modeling was carried out using an ontology, and mapping was done with Health Level 7 Fast Healthcare Interoperability Resources (HL7 FHIR) and Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT; International Health Terminology Standards Development Organisation). Once the CDSS was developed, validation was carried out by using a retrospective case study. Results: This project was funded in January 2015 and approved by the Virgen del Rocio University Hospital ethics committee on November 24, 2015. Two different tasks were carried out to test the functioning of the tool. First, retrospective data from a real patient who met the inclusion criteria were used. Second, the analysis of an adoption model was performed through the study of the requirements and characteristics that a CDSS must meet in order to be well accepted and used by health professionals. The results are favorable and allow the proposed research to continue to the next phase. Conclusions: An ontology-based CDSS was successfully designed, developed, and validated. However, in future work, validation in a real environment should be performed to ensure the tool is usable and reliable. UR - https://formative.jmir.org/2022/8/e27990 UR - http://dx.doi.org/10.2196/27990 UR - http://www.ncbi.nlm.nih.gov/pubmed/35916719 ID - info:doi/10.2196/27990 ER - TY - JOUR AU - Black, Bell Georgia AU - Bhuiya, Afsana AU - Friedemann Smith, Claire AU - Hirst, Yasemin AU - Nicholson, David Brian PY - 2022/8/1 TI - Harnessing the Electronic Health Care Record to Optimize Patient Safety in Primary Care: Framework for Evaluating e?Safety-Netting Tools JO - JMIR Med Inform SP - e35726 VL - 10 IS - 8 KW - primary care KW - patient safety KW - electronic health record KW - safety KW - optimize KW - framework KW - evaluation KW - tool KW - diagnostic KW - uncertainty KW - management KW - netting KW - software KW - criteria UR - https://medinform.jmir.org/2022/8/e35726 UR - http://dx.doi.org/10.2196/35726 UR - http://www.ncbi.nlm.nih.gov/pubmed/35916722 ID - info:doi/10.2196/35726 ER - TY - JOUR AU - Chen, Pei-Fu AU - Chen, Kuan-Chih AU - Liao, Wei-Chih AU - Lai, Feipei AU - He, Tai-Liang AU - Lin, Sheng-Che AU - Chen, Wei-Jen AU - Yang, Chi-Yu AU - Lin, Yu-Cheng AU - Tsai, I-Chang AU - Chiu, Chi-Hao AU - Chang, Shu-Chih AU - Hung, Fang-Ming PY - 2022/6/29 TI - Automatic International Classification of Diseases Coding System: Deep Contextualized Language Model With Rule-Based Approaches JO - JMIR Med Inform SP - e37557 VL - 10 IS - 6 KW - deep learning KW - International Classification of Diseases KW - medical records KW - multilabel text classification KW - natural language processing KW - coding system KW - algorithm KW - electronic health record KW - data mining N2 - Background: The tenth revision of the International Classification of Diseases (ICD-10) is widely used for epidemiological research and health management. The clinical modification (CM) and procedure coding system (PCS) of ICD-10 were developed to describe more clinical details with increasing diagnosis and procedure codes and applied in disease-related groups for reimbursement. The expansion of codes made the coding time-consuming and less accurate. The state-of-the-art model using deep contextual word embeddings was used for automatic multilabel text classification of ICD-10. In addition to input discharge diagnoses (DD), the performance can be improved by appropriate preprocessing methods for the text from other document types, such as medical history, comorbidity and complication, surgical method, and special examination. Objective: This study aims to establish a contextual language model with rule-based preprocessing methods to develop the model for ICD-10 multilabel classification. Methods: We retrieved electronic health records from a medical center. We first compared different word embedding methods. Second, we compared the preprocessing methods using the best-performing embeddings. We compared biomedical bidirectional encoder representations from transformers (BioBERT), clinical generalized autoregressive pretraining for language understanding (Clinical XLNet), label tree-based attention-aware deep model for high-performance extreme multilabel text classification (AttentionXLM), and word-to-vector (Word2Vec) to predict ICD-10-CM. To compare different preprocessing methods for ICD-10-CM, we included DD, medical history, and comorbidity and complication as inputs. We compared the performance of ICD-10-CM prediction using different preprocesses, including definition training, external cause code removal, number conversion, and combination code filtering. For the ICD-10 PCS, the model was trained using different combinations of DD, surgical method, and key words of special examination. The micro F1 score and the micro area under the receiver operating characteristic curve were used to compare the model?s performance with that of different preprocessing methods. Results: BioBERT had an F1 score of 0.701 and outperformed other models such as Clinical XLNet, AttentionXLM, and Word2Vec. For the ICD-10-CM, the model had an F1 score that significantly increased from 0.749 (95% CI 0.744-0.753) to 0.769 (95% CI 0.764-0.773) with the ICD-10 definition training, external cause code removal, number conversion, and combination code filter. For the ICD-10-PCS, the model had an F1 score that significantly increased from 0.670 (95% CI 0.663-0.678) to 0.726 (95% CI 0.719-0.732) with a combination of discharge diagnoses, surgical methods, and key words of special examination. With our preprocessing methods, the model had the highest area under the receiver operating characteristic curve of 0.853 (95% CI 0.849-0.855) and 0.831 (95% CI 0.827-0.834) for ICD-10-CM and ICD-10-PCS, respectively. Conclusions: The performance of our model with the pretrained contextualized language model and rule-based preprocessing method is better than that of the state-of-the-art model for ICD-10-CM or ICD-10-PCS. This study highlights the importance of rule-based preprocessing methods based on coder coding rules. UR - https://medinform.jmir.org/2022/6/e37557 UR - http://dx.doi.org/10.2196/37557 UR - http://www.ncbi.nlm.nih.gov/pubmed/35767353 ID - info:doi/10.2196/37557 ER - TY - JOUR AU - Chatterjee, Ayan AU - Prinz, Andreas PY - 2022/6/23 TI - Personalized Recommendations for Physical Activity e-Coaching (OntoRecoModel): Ontological Modeling JO - JMIR Med Inform SP - e33847 VL - 10 IS - 6 KW - descriptive logic KW - ontology KW - e-coach KW - reasoning KW - recommendation generation N2 - Background: Automatic e-coaching may motivate individuals to lead a healthy lifestyle with early health risk prediction, personalized recommendation generation, and goal evaluation. Multiple studies have reported on uninterrupted and automatic monitoring of behavioral aspects (such as sedentary time, amount, and type of physical activity); however, e-coaching and personalized feedback techniques are still in a nascent stage. Current intelligent coaching strategies are mostly based on the handcrafted string messages that rarely individualize to each user?s needs, context, and preferences. Therefore, more realistic, flexible, practical, sophisticated, and engaging strategies are needed to model personalized recommendations. Objective: This study aims to design and develop an ontology to model personalized recommendation message intent, components (such as suggestion, feedback, argument, and follow-ups), and contents (such as spatial and temporal content and objects relevant to perform the recommended activities). A reasoning technique will help to discover implied knowledge from the proposed ontology. Furthermore, recommendation messages can be classified into different categories in the proposed ontology. Methods: The ontology was created using Protégé (version 5.5.0) open-source software. We used the Java-based Jena Framework (version 3.16) to build a semantic web application as a proof of concept, which included Resource Description Framework application programming interface, World Wide Web Consortium Web Ontology Language application programming interface, native tuple database, and SPARQL Protocol and Resource Description Framework Query Language query engine. The HermiT (version 1.4.3.x) ontology reasoner available in Protégé 5.x implemented the logical and structural consistency of the proposed ontology. To verify the proposed ontology model, we simulated data for 8 test cases. The personalized recommendation messages were generated based on the processing of personal activity data in combination with contextual weather data and personal preference data. The developed ontology was processed using a query engine against a rule base to generate personalized recommendations. Results: The proposed ontology was implemented in automatic activity coaching to generate and deliver meaningful, personalized lifestyle recommendations. The ontology can be visualized using OWLViz and OntoGraf. In addition, we developed an ontology verification module that behaves similar to a rule-based decision support system to analyze the generation and delivery of personalized recommendation messages following a logical structure. Conclusions: This study led to the creation of a meaningful ontology to generate and model personalized recommendation messages for physical activity coaching. UR - https://medinform.jmir.org/2022/6/e33847 UR - http://dx.doi.org/10.2196/33847 UR - http://www.ncbi.nlm.nih.gov/pubmed/35737439 ID - info:doi/10.2196/33847 ER - TY - JOUR AU - Singhal, Richa AU - Lukose, Rachel AU - Carr, Gwenyth AU - Moktar, Afsoon AU - Gonzales-Urday, Lucia Ana AU - Rouchka, C. Eric AU - Vajravelu, N. Bathri PY - 2022/6/17 TI - Differential Expression of Long Noncoding RNAs in Murine Myoblasts After Short Hairpin RNA-Mediated Dysferlin Silencing In Vitro: Microarray Profiling JO - JMIR Bioinform Biotech SP - e33186 VL - 3 IS - 1 KW - dysferlinopathy KW - long noncoding RNAs KW - lncRNA KW - abnormal expression KW - muscular dystrophy KW - limb-girdle muscular dystrophy 2B KW - LGMD-2B KW - messenger RNA KW - mRNA KW - quantitative real-time polymerase chain reaction KW - qRT-PCR KW - gene ontology KW - bioinformatics KW - transcription KW - noncoding RNA KW - protein expression N2 - Background: Long noncoding RNAs (lncRNAs) are noncoding RNA transcripts greater than 200 nucleotides in length and are known to play a role in regulating the transcription of genes involved in vital cellular functions. We hypothesized the disease process in dysferlinopathy is linked to an aberrant expression of lncRNAs and messenger RNAs (mRNAs). Objective: In this study, we compared the lncRNA and mRNA expression profiles between wild-type and dysferlin-deficient murine myoblasts (C2C12 cells). Methods: LncRNA and mRNA expression profiling were performed using a microarray. Several lncRNAs with differential expression were validated using quantitative real-time polymerase chain reaction. Gene Ontology (GO) analysis was performed to understand the functional role of the differentially expressed mRNAs. Further bioinformatics analysis was used to explore the potential function, lncRNA-mRNA correlation, and potential targets of the differentially expressed lncRNAs. Results: We found 3195 lncRNAs and 1966 mRNAs that were differentially expressed. The chromosomal distribution of the differentially expressed lncRNAs and mRNAs was unequal, with chromosome 2 having the highest number of lncRNAs and chromosome 7 having the highest number of mRNAs that were differentially expressed. Pathway analysis of the differentially expressed genes indicated the involvement of several signaling pathways including PI3K-Akt, Hippo, and pathways regulating the pluripotency of stem cells. The differentially expressed genes were also enriched for the GO terms, developmental process and muscle system process. Network analysis identified 8 statistically significant (P<.05) network objects from the upregulated lncRNAs and 3 statistically significant network objects from the downregulated lncRNAs. Conclusions: Our results thus far imply that dysferlinopathy is associated with an aberrant expression of multiple lncRNAs, many of which may have a specific function in the disease process. GO terms and network analysis suggest a muscle-specific role for these lncRNAs. To elucidate the specific roles of these abnormally expressed noncoding RNAs, further studies engineering their expression are required. UR - https://bioinform.jmir.org/2022/1/e33186 UR - http://dx.doi.org/10.2196/33186 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/33186 ER - TY - JOUR AU - Li, Shicheng AU - Deng, Lizong AU - Zhang, Xu AU - Chen, Luming AU - Yang, Tao AU - Qi, Yifan AU - Jiang, Taijiao PY - 2022/6/3 TI - Deep Phenotyping of Chinese Electronic Health Records by Recognizing Linguistic Patterns of Phenotypic Narratives With a Sequence Motif Discovery Tool: Algorithm Development and Validation JO - J Med Internet Res SP - e37213 VL - 24 IS - 6 KW - deep phenotyping KW - Chinese EHRs KW - linguistic pattern KW - motif discovery KW - pattern recognition N2 - Background: Phenotype information in electronic health records (EHRs) is mainly recorded in unstructured free text, which cannot be directly used for clinical research. EHR-based deep-phenotyping methods can structure phenotype information in EHRs with high fidelity, making it the focus of medical informatics. However, developing a deep-phenotyping method for non-English EHRs (ie, Chinese EHRs) is challenging. Although numerous EHR resources exist in China, fine-grained annotation data that are suitable for developing deep-phenotyping methods are limited. It is challenging to develop a deep-phenotyping method for Chinese EHRs in such a low-resource scenario. Objective: In this study, we aimed to develop a deep-phenotyping method with good generalization ability for Chinese EHRs based on limited fine-grained annotation data. Methods: The core of the methodology was to identify linguistic patterns of phenotype descriptions in Chinese EHRs with a sequence motif discovery tool and perform deep phenotyping of Chinese EHRs by recognizing linguistic patterns in free text. Specifically, 1000 Chinese EHRs were manually annotated based on a fine-grained information model, PhenoSSU (Semantic Structured Unit of Phenotypes). The annotation data set was randomly divided into a training set (n=700, 70%) and a testing set (n=300, 30%). The process for mining linguistic patterns was divided into three steps. First, free text in the training set was encoded as single-letter sequences (P: phenotype, A: attribute). Second, a biological sequence analysis tool?MEME (Multiple Expectation Maximums for Motif Elicitation)?was used to identify motifs in the single-letter sequences. Finally, the identified motifs were reduced to a series of regular expressions representing linguistic patterns of PhenoSSU instances in Chinese EHRs. Based on the discovered linguistic patterns, we developed a deep-phenotyping method for Chinese EHRs, including a deep learning?based method for named entity recognition and a pattern recognition?based method for attribute prediction. Results: In total, 51 sequence motifs with statistical significance were mined from 700 Chinese EHRs in the training set and were combined into six regular expressions. It was found that these six regular expressions could be learned from a mean of 134 (SD 9.7) annotated EHRs in the training set. The deep-phenotyping algorithm for Chinese EHRs could recognize PhenoSSU instances with an overall accuracy of 0.844 on the test set. For the subtask of entity recognition, the algorithm achieved an F1 score of 0.898 with the Bidirectional Encoder Representations from Transformers?bidirectional long short-term memory and conditional random field model; for the subtask of attribute prediction, the algorithm achieved a weighted accuracy of 0.940 with the linguistic pattern?based method. Conclusions: We developed a simple but effective strategy to perform deep phenotyping of Chinese EHRs with limited fine-grained annotation data. Our work will promote the second use of Chinese EHRs and give inspiration to other non?English-speaking countries. UR - https://www.jmir.org/2022/6/e37213 UR - http://dx.doi.org/10.2196/37213 UR - http://www.ncbi.nlm.nih.gov/pubmed/35657661 ID - info:doi/10.2196/37213 ER - TY - JOUR AU - Gruendner, Julian AU - Deppenwiese, Noemi AU - Folz, Michael AU - Köhler, Thomas AU - Kroll, Björn AU - Prokosch, Hans-Ulrich AU - Rosenau, Lorenz AU - Rühle, Mathias AU - Scheidl, Marc-Anton AU - Schüttler, Christina AU - Sedlmayr, Brita AU - Twrdik, Alexander AU - Kiel, Alexander AU - Majeed, W. Raphael PY - 2022/5/25 TI - The Architecture of a Feasibility Query Portal for Distributed COVID-19 Fast Healthcare Interoperability Resources (FHIR) Patient Data Repositories: Design and Implementation Study JO - JMIR Med Inform SP - e36709 VL - 10 IS - 5 KW - federated feasibility queries KW - FHIR KW - distributed analysis KW - feasibility study KW - HL7 FHIR KW - FHIR Search KW - CQL KW - COVID-19 KW - pandemic KW - health data KW - query KW - patient data KW - consensus data set KW - medical informatics KW - Fast Healthcare Interoperability Resources N2 - Background: An essential step in any medical research project after identifying the research question is to determine if there are sufficient patients available for a study and where to find them. Pursuing digital feasibility queries on available patient data registries has proven to be an excellent way of reusing existing real-world data sources. To support multicentric research, these feasibility queries should be designed and implemented to run across multiple sites and securely access local data. Working across hospitals usually involves working with different data formats and vocabularies. Recently, the Fast Healthcare Interoperability Resources (FHIR) standard was developed by Health Level Seven to address this concern and describe patient data in a standardized format. The Medical Informatics Initiative in Germany has committed to this standard and created data integration centers, which convert existing data into the FHIR format at each hospital. This partially solves the interoperability problem; however, a distributed feasibility query platform for the FHIR standard is still missing. Objective: This study described the design and implementation of the components involved in creating a cross-hospital feasibility query platform for researchers based on FHIR resources. This effort was part of a large COVID-19 data exchange platform and was designed to be scalable for a broad range of patient data. Methods: We analyzed and designed the abstract components necessary for a distributed feasibility query. This included a user interface for creating the query, backend with an ontology and terminology service, middleware for query distribution, and FHIR feasibility query execution service. Results: We implemented the components described in the Methods section. The resulting solution was distributed to 33 German university hospitals. The functionality of the comprehensive network infrastructure was demonstrated using a test data set based on the German Corona Consensus Data Set. A performance test using specifically created synthetic data revealed the applicability of our solution to data sets containing millions of FHIR resources. The solution can be easily deployed across hospitals and supports feasibility queries, combining multiple inclusion and exclusion criteria using standard Health Level Seven query languages such as Clinical Quality Language and FHIR Search. Developing a platform based on multiple microservices allowed us to create an extendable platform and support multiple Health Level Seven query languages and middleware components to allow integration with future directions of the Medical Informatics Initiative. Conclusions: We designed and implemented a feasibility platform for distributed feasibility queries, which works directly on FHIR-formatted data and distributed it across 33 university hospitals in Germany. We showed that developing a feasibility platform directly on the FHIR standard is feasible. UR - https://medinform.jmir.org/2022/5/e36709 UR - http://dx.doi.org/10.2196/36709 UR - http://www.ncbi.nlm.nih.gov/pubmed/35486893 ID - info:doi/10.2196/36709 ER - TY - JOUR AU - Abaza, Haitham AU - Kadioglu, Dennis AU - Martin, Simona AU - Papadopoulou, Andri AU - dos Santos Vieira, Bruna AU - Schaefer, Franz AU - Storf, Holger PY - 2022/5/20 TI - Domain-Specific Common Data Elements for Rare Disease Registration: Conceptual Approach of a European Joint Initiative Toward Semantic Interoperability in Rare Disease Research JO - JMIR Med Inform SP - e32158 VL - 10 IS - 5 KW - semantic interoperability KW - common data elements KW - standardization KW - data collection KW - data discoverability KW - rare diseases KW - EJP RD KW - EU RD Platform KW - ERNs KW - FAIRification KW - health infrastructure KW - industry KW - medical informatics KW - health platforms KW - health registries KW - health and research platforms KW - health domains N2 - Background: With hundreds of registries across Europe, rare diseases (RDs) suffer from fragmented knowledge, expertise, and research. A joint initiative of the European Commission Joint Research Center and its European Platform on Rare Disease Registration (EU RD Platform), the European Reference Networks (ERNs), and the European Joint Programme on Rare Diseases (EJP RD) was launched in 2020. The purpose was to extend the set of common data elements (CDEs) for RD registration by defining domain-specific CDEs (DCDEs). Objective: This study aims to introduce and assess the feasibility of the concept of a joint initiative that unites the efforts of the European Platform on Rare Disease Registration Platform, ERNs, and European Joint Programme on Rare Diseases toward extending RD CDEs, aiming to improve the semantic interoperability of RD registries and enhance the quality of RD research. Methods: A joint conference was conducted in December 2020. All 24 ERNs were invited. Before the conference, a survey was communicated to all ERNs, proposing 18 medical domains and requesting them to identify highly relevant choices. After the conference, a 3-phase plan for defining and modeling DCDEs was drafted. Expected outcomes included harmonized lists of DCDEs. Results: All ERNs attended the conference. The survey results indicated that genetic, congenital, pediatric, and cancer were the most overlapping domains. Accordingly, the proposed list was reorganized into 10 domain groups and recommunicated to all ERNs, aiming at a smaller number of domains. Conclusions: The approach described for defining DCDEs appears to be feasible. However, it remains dynamic and should be repeated regularly based on arising research needs. UR - https://medinform.jmir.org/2022/5/e32158 UR - http://dx.doi.org/10.2196/32158 UR - http://www.ncbi.nlm.nih.gov/pubmed/35594066 ID - info:doi/10.2196/32158 ER - TY - JOUR AU - Rosenau, Lorenz AU - Majeed, W. Raphael AU - Ingenerf, Josef AU - Kiel, Alexander AU - Kroll, Björn AU - Köhler, Thomas AU - Prokosch, Hans-Ulrich AU - Gruendner, Julian PY - 2022/4/27 TI - Generation of a Fast Healthcare Interoperability Resources (FHIR)-based Ontology for Federated Feasibility Queries in the Context of COVID-19: Feasibility Study JO - JMIR Med Inform SP - e35789 VL - 10 IS - 4 KW - federated queries KW - feasibility study KW - Fast Healthcare Interoperability Resource KW - FHIR Search KW - CQL KW - ontology KW - terminology server KW - query KW - feasibility KW - FHIR KW - terminology KW - development KW - COVID-19 KW - automation KW - user interface KW - map KW - input KW - hospital KW - data KW - Germany KW - accessibility KW - harmonized N2 - Background: The COVID-19 pandemic highlighted the importance of making research data from all German hospitals available to scientists to respond to current and future pandemics promptly. The heterogeneous data originating from proprietary systems at hospitals' sites must be harmonized and accessible. The German Corona Consensus Dataset (GECCO) specifies how data for COVID-19 patients will be standardized in Fast Healthcare Interoperability Resources (FHIR) profiles across German hospitals. However, given the complexity of the FHIR standard, the data harmonization is not sufficient to make the data accessible. A simplified visual representation is needed to reduce the technical burden, while allowing feasibility queries. Objective: This study investigates how a search ontology can be automatically generated using FHIR profiles and a terminology server. Furthermore, it describes how this ontology can be used in a user interface (UI) and how a mapping and a terminology tree created together with the ontology can translate user input into FHIR queries. Methods: We used the FHIR profiles from the GECCO data set combined with a terminology server to generate an ontology and the required mapping files for the translation. We analyzed the profiles and identified search criteria for the visual representation. In this process, we reduced the complex profiles to code value pairs for improved usability. We enriched our ontology with the necessary information to display it in a UI. We also developed an intermediate query language to transform the queries from the UI to federated FHIR requests. Separation of concerns resulted in discrepancies between the criteria used in the intermediate query format and the target query language. Therefore, a mapping was created to reintroduce all information relevant for creating the query in its target language. Further, we generated a tree representation of the ontology hierarchy, which allows resolving child concepts in the process. Results: In the scope of this project, 82 (99%) of 83 elements defined in the GECCO profile were successfully implemented. We verified our solution based on an independently developed test patient. A discrepancy between the test data and the criteria was found in 6 cases due to different versions used to generate the test data and the UI profiles, the support for specific code systems, and the evaluation of postcoordinated Systematized Nomenclature of Medicine (SNOMED) codes. Our results highlight the need for governance mechanisms for version changes, concept mapping between values from different code systems encoding the same concept, and support for different unit dimensions. Conclusions: We developed an automatic process to generate ontology and mapping files for FHIR-formatted data. Our tests found that this process works for most of our chosen FHIR profile criteria. The process established here works directly with FHIR profiles and a terminology server, making it extendable to other FHIR profiles and demonstrating that automatic ontology generation on FHIR profiles is feasible. UR - https://medinform.jmir.org/2022/4/e35789 UR - http://dx.doi.org/10.2196/35789 UR - http://www.ncbi.nlm.nih.gov/pubmed/35380548 ID - info:doi/10.2196/35789 ER - TY - JOUR AU - Sun, Yuanyuan AU - Gao, Dongping AU - Shen, Xifeng AU - Li, Meiting AU - Nan, Jiale AU - Zhang, Weining PY - 2022/4/21 TI - Multi-Label Classification in Patient-Doctor Dialogues With the RoBERTa-WWM-ext + CNN (Robustly Optimized Bidirectional Encoder Representations From Transformers Pretraining Approach With Whole Word Masking Extended Combining a Convolutional Neural Network) Model: Named Entity Study JO - JMIR Med Inform SP - e35606 VL - 10 IS - 4 KW - online consultation KW - named entity KW - automatic classification KW - ERNIE KW - Enhanced Representation through Knowledge Integration KW - BERT KW - Bidirectional Encoder Representations from Transformers KW - machine learning KW - neural network KW - model KW - China KW - Chinese KW - classification KW - patient-physician dialogue KW - patient doctor dialogue KW - semantics KW - natural language processing N2 - Background: With the prevalence of online consultation, many patient-doctor dialogues have accumulated, which, in an authentic language environment, are of significant value to the research and development of intelligent question answering and automated triage in recent natural language processing studies. Objective: The purpose of this study was to design a front-end task module for the network inquiry of intelligent medical services. Through the study of automatic labeling of real doctor-patient dialogue text on the internet, a method of identifying the negative and positive entities of dialogues with higher accuracy has been explored. Methods: The data set used for this study was from the Spring Rain Doctor internet online consultation, which was downloaded from the official data set of Alibaba Tianchi Lab. We proposed a composite abutting joint model, which was able to automatically classify the types of clinical finding entities into the following 4 attributes: positive, negative, other, and empty. We adapted a downstream architecture in Chinese Robustly Optimized Bidirectional Encoder Representations from Transformers Pretraining Approach (RoBERTa) with whole word masking (WWM) extended (RoBERTa-WWM-ext) combining a text convolutional neural network (CNN). We used RoBERTa-WWM-ext to express sentence semantics as a text vector and then extracted the local features of the sentence through the CNN, which was our new fusion model. To verify its knowledge learning ability, we chose Enhanced Representation through Knowledge Integration (ERNIE), original Bidirectional Encoder Representations from Transformers (BERT), and Chinese BERT with WWM to perform the same task, and then compared the results. Precision, recall, and macro-F1 were used to evaluate the performance of the methods. Results: We found that the ERNIE model, which was trained with a large Chinese corpus, had a total score (macro-F1) of 65.78290014, while BERT and BERT-WWM had scores of 53.18247117 and 69.2795315, respectively. Our composite abutting joint model (RoBERTa-WWM-ext + CNN) had a macro-F1 value of 70.55936311, showing that our model outperformed the other models in the task. Conclusions: The accuracy of the original model can be greatly improved by giving priority to WWM and replacing the word-based mask with unit to classify and label medical entities. Better results can be obtained by effectively optimizing the downstream tasks of the model and the integration of multiple models later on. The study findings contribute to the translation of online consultation information into machine-readable information. UR - https://medinform.jmir.org/2022/4/e35606 UR - http://dx.doi.org/10.2196/35606 UR - http://www.ncbi.nlm.nih.gov/pubmed/35451969 ID - info:doi/10.2196/35606 ER - TY - JOUR AU - Bae, Ho Jung AU - Han, Wook Hyun AU - Yang, Young Sun AU - Song, Gyuseon AU - Sa, Soonok AU - Chung, Eun Goh AU - Seo, Yeon Ji AU - Jin, Hyo Eun AU - Kim, Heecheon AU - An, DongUk PY - 2022/4/15 TI - Natural Language Processing for Assessing Quality Indicators in Free-Text Colonoscopy and Pathology Reports: Development and Usability Study JO - JMIR Med Inform SP - e35257 VL - 10 IS - 4 KW - natural language processing KW - colonoscopy KW - adenoma KW - endoscopy N2 - Background: Manual data extraction of colonoscopy quality indicators is time and labor intensive. Natural language processing (NLP), a computer-based linguistics technique, can automate the extraction of important clinical information, such as adverse events, from unstructured free-text reports. NLP information extraction can facilitate the optimization of clinical work by helping to improve quality control and patient management. Objective: We developed an NLP pipeline to analyze free-text colonoscopy and pathology reports and evaluated its ability to automatically assess adenoma detection rate (ADR), sessile serrated lesion detection rate (SDR), and postcolonoscopy surveillance intervals. Methods: The NLP tool for extracting colonoscopy quality indicators was developed using a data set of 2000 screening colonoscopy reports from a single health care system, with an associated 1425 pathology reports. The NLP system was then tested on a data set of 1000 colonoscopy reports and its performance was compared with that of 5 human annotators. Additionally, data from 54,562 colonoscopies performed between 2010 and 2019 were analyzed using the NLP pipeline. Results: The NLP pipeline achieved an overall accuracy of 0.99-1.00 for identifying polyp subtypes, 0.99-1.00 for identifying the anatomical location of polyps, and 0.98 for counting the number of neoplastic polyps. The NLP pipeline achieved performance similar to clinical experts for assessing ADR, SDR, and surveillance intervals. NLP analysis of a 10-year colonoscopy data set identified great individual variance in colonoscopy quality indicators among 25 endoscopists. Conclusions: The NLP pipeline could accurately extract information from colonoscopy and pathology reports and demonstrated clinical efficacy for assessing ADR, SDR, and surveillance intervals in these reports. Implementation of the system enabled automated analysis and feedback on quality indicators, which could motivate endoscopists to improve the quality of their performance and improve clinical decision-making in colorectal cancer screening programs. UR - https://medinform.jmir.org/2022/4/e35257 UR - http://dx.doi.org/10.2196/35257 UR - http://www.ncbi.nlm.nih.gov/pubmed/35436226 ID - info:doi/10.2196/35257 ER - TY - JOUR AU - Falissard, Louis AU - Morgand, Claire AU - Ghosn, Walid AU - Imbaud, Claire AU - Bounebache, Karim AU - Rey, Grégoire PY - 2022/4/11 TI - Neural Translation and Automated Recognition of ICD-10 Medical Entities From Natural Language: Model Development and Performance Assessment JO - JMIR Med Inform SP - e26353 VL - 10 IS - 4 KW - machine learning KW - deep learning KW - machine translation KW - mortality statistics KW - automated medical entity recognition KW - ICD-10 coding N2 - Background: The recognition of medical entities from natural language is a ubiquitous problem in the medical field, with applications ranging from medical coding to the analysis of electronic health data for public health. It is, however, a complex task usually requiring human expert intervention, thus making it expansive and time-consuming. Recent advances in artificial intelligence, specifically the rise of deep learning methods, have enabled computers to make efficient decisions on a number of complex problems, with the notable example of neural sequence models and their powerful applications in natural language processing. However, they require a considerable amount of data to learn from, which is typically their main limiting factor. The Centre for Epidemiology on Medical Causes of Death (CépiDc) stores an exhaustive database of death certificates at the French national scale, amounting to several millions of natural language examples provided with their associated human-coded medical entities available to the machine learning practitioner. Objective: The aim of this paper was to investigate the application of deep neural sequence models to the problem of medical entity recognition from natural language. Methods: The investigated data set included every French death certificate from 2011 to 2016. These certificates contain information such as the subject?s age, the subject?s gender, and the chain of events leading to his or her death, both in French and encoded as International Statistical Classification of Diseases and Related Health Problems, Tenth Revision (ICD-10) medical entities, for a total of around 3 million observations in the data set. The task of automatically recognizing ICD-10 medical entities from the French natural language?based chain of events leading to death was then formulated as a type of predictive modeling problem known as a sequence-to-sequence modeling problem. A deep neural network?based model, known as the Transformer, was then slightly adapted and fit to the data set. Its performance was then assessed on an external data set and compared to the current state-of-the-art approach. CIs for derived measurements were estimated via bootstrapping. Results: The proposed approach resulted in an F-measure value of 0.952 (95% CI 0.946-0.957), which constitutes a significant improvement over the current state-of-the-art approach and its previously reported F-measure value of 0.825 as assessed on a comparable data set. Such an improvement makes possible a whole field of new applications, from nosologist-level automated coding to temporal harmonization of death statistics. Conclusions: This paper shows that a deep artificial neural network can directly learn from voluminous data sets in order to identify complex relationships between natural language and medical entities, without any explicit prior knowledge. Although not entirely free from mistakes, the derived model constitutes a powerful tool for automated coding of medical entities from medical language with promising potential applications. UR - https://medinform.jmir.org/2022/4/e26353 UR - http://dx.doi.org/10.2196/26353 UR - http://www.ncbi.nlm.nih.gov/pubmed/35404262 ID - info:doi/10.2196/26353 ER - TY - JOUR AU - Huang, Zonghai AU - Miao, Jiaqing AU - Chen, Ju AU - Zhong, Yanmei AU - Yang, Simin AU - Ma, Yiyi AU - Wen, Chuanbiao PY - 2022/4/6 TI - A Traditional Chinese Medicine Syndrome Classification Model Based on Cross-Feature Generation by Convolution Neural Network: Model Development and Validation JO - JMIR Med Inform SP - e29290 VL - 10 IS - 4 KW - intelligent syndrome differentiation KW - cross-FGCNN KW - TCM N2 - Background: Nowadays, intelligent medicine is gaining widespread attention, and great progress has been made in Western medicine with the help of artificial intelligence to assist in decision making. Compared with Western medicine, traditional Chinese medicine (TCM) involves selecting the specific treatment method, prescription, and medication based on the dialectical results of each patient?s symptoms. For this reason, the development of a TCM-assisted decision-making system has lagged. Treatment based on syndrome differentiation is the core of TCM treatment; TCM doctors can dialectically classify diseases according to patients? symptoms and optimize treatment in time. Therefore, the essence of a TCM-assisted decision-making system is a TCM intelligent, dialectical algorithm. Symptoms stored in electronic medical records are mostly associated with patients? diseases; however, symptoms of TCM are mostly subjectively identified. In general electronic medical records, there are many missing values. TCM medical records, in which symptoms tend to cause high-dimensional sparse data, reduce algorithm accuracy. Objective: This study aims to construct an algorithm model compatible for the multidimensional, highly sparse, and multiclassification task of TCM syndrome differentiation, so that it can be effectively applied to the intelligent dialectic of different diseases. Methods: The relevant terms in electronic medical records were standardized with respect to symptoms and evidence-based criteria of TCM. We structuralized case data based on the classification of different symptoms and physical signs according to the 4 diagnostic examinations in TCM diagnosis. A novel cross-feature generation by convolution neural network model performed evidence-based recommendations based on the input embedded, structured medical record data. Results: The data set included 5273 real dysmenorrhea cases from the Sichuan TCM big data management platform and the Chinese literature database, which were embedded into 60 fields after being structured and standardized. The training set and test set were randomly constructed in a ratio of 3:1. For the classification of different syndrome types, compared with 6 traditional, intelligent dialectical models and 3 click-through-rate models, the new model showed a good generalization ability and good classification effect. The comprehensive accuracy rate reached 96.21%. Conclusions: The main contribution of this study is the construction of a new intelligent dialectical model combining the characteristics of TCM by treating intelligent dialectics as a high-dimensional sparse vector classification task. Owing to the standardization of the input symptoms, all the common symptoms of TCM are covered, and the model can differentiate the symptoms with a variety of missing values. Therefore, with the continuous improvement of disease data sets, this model has the potential to be applied to the dialectical classification of different diseases in TCM. UR - https://medinform.jmir.org/2022/4/e29290 UR - http://dx.doi.org/10.2196/29290 UR - http://www.ncbi.nlm.nih.gov/pubmed/35384854 ID - info:doi/10.2196/29290 ER - TY - JOUR AU - Mitchell, Ross Joseph AU - Szepietowski, Phillip AU - Howard, Rachel AU - Reisman, Phillip AU - Jones, D. Jennie AU - Lewis, Patricia AU - Fridley, L. Brooke AU - Rollison, E. Dana PY - 2022/3/23 TI - A Question-and-Answer System to Extract Data From Free-Text Oncological Pathology Reports (CancerBERT Network): Development Study JO - J Med Internet Res SP - e27210 VL - 24 IS - 3 KW - natural language processing KW - NLP KW - BERT KW - transformer KW - pathology KW - ICD-O-3 KW - deep learning KW - cancer N2 - Background: Information in pathology reports is critical for cancer care. Natural language processing (NLP) systems used to extract information from pathology reports are often narrow in scope or require extensive tuning. Consequently, there is growing interest in automated deep learning approaches. A powerful new NLP algorithm, bidirectional encoder representations from transformers (BERT), was published in late 2018. BERT set new performance standards on tasks as diverse as question answering, named entity recognition, speech recognition, and more. Objective: The aim of this study is to develop a BERT-based system to automatically extract detailed tumor site and histology information from free-text oncological pathology reports. Methods: We pursued three specific aims: extract accurate tumor site and histology descriptions from free-text pathology reports, accommodate the diverse terminology used to indicate the same pathology, and provide accurate standardized tumor site and histology codes for use by downstream applications. We first trained a base language model to comprehend the technical language in pathology reports. This involved unsupervised learning on a training corpus of 275,605 electronic pathology reports from 164,531 unique patients that included 121 million words. Next, we trained a question-and-answer (Q&A) model that connects a Q&A layer to the base pathology language model to answer pathology questions. Our Q&A system was designed to search for the answers to two predefined questions in each pathology report: What organ contains the tumor? and What is the kind of tumor or carcinoma? This involved supervised training on 8197 pathology reports, each with ground truth answers to these 2 questions determined by certified tumor registrars. The data set included 214 tumor sites and 193 histologies. The tumor site and histology phrases extracted by the Q&A model were used to predict International Classification of Diseases for Oncology, Third Edition (ICD-O-3), site and histology codes. This involved fine-tuning two additional BERT models: one to predict site codes and another to predict histology codes. Our final system includes a network of 3 BERT-based models. We call this CancerBERT network (caBERTnet). We evaluated caBERTnet using a sequestered test data set of 2050 pathology reports with ground truth answers determined by certified tumor registrars. Results: caBERTnet?s accuracies for predicting group-level site and histology codes were 93.53% (1895/2026) and 97.6% (1993/2042), respectively. The top 5 accuracies for predicting fine-grained ICD-O-3 site and histology codes with 5 or more samples each in the training data set were 92.95% (1794/1930) and 96.01% (1853/1930), respectively. Conclusions: We have developed an NLP system that outperforms existing algorithms at predicting ICD-O-3 codes across an extensive range of tumor sites and histologies. Our new system could help reduce treatment delays, increase enrollment in clinical trials of new therapies, and improve patient outcomes. UR - https://www.jmir.org/2022/3/e27210 UR - http://dx.doi.org/10.2196/27210 UR - http://www.ncbi.nlm.nih.gov/pubmed/35319481 ID - info:doi/10.2196/27210 ER - TY - JOUR AU - Almowil, Zahra AU - Zhou, Shang-Ming AU - Brophy, Sinead AU - Croxall, Jodie PY - 2022/3/15 TI - Concept Libraries for Repeatable and Reusable Research: Qualitative Study Exploring the Needs of Users JO - JMIR Hum Factors SP - e31021 VL - 9 IS - 1 KW - electronic health records KW - record linkage KW - reproducible research KW - clinical codes KW - concept libraries N2 - Background: Big data research in the field of health sciences is hindered by a lack of agreement on how to identify and define different conditions and their medications. This means that researchers and health professionals often have different phenotype definitions for the same condition. This lack of agreement makes it difficult to compare different study findings and hinders the ability to conduct repeatable and reusable research. Objective: This study aims to examine the requirements of various users, such as researchers, clinicians, machine learning experts, and managers, in the development of a data portal for phenotypes (a concept library). Methods: This was a qualitative study using interviews and focus group discussion. One-to-one interviews were conducted with researchers, clinicians, machine learning experts, and senior research managers in health data science (N=6) to explore their specific needs in the development of a concept library. In addition, a focus group discussion with researchers (N=14) working with the Secured Anonymized Information Linkage databank, a national eHealth data linkage infrastructure, was held to perform a SWOT (strengths, weaknesses, opportunities, and threats) analysis for the phenotyping system and the proposed concept library. The interviews and focus group discussion were transcribed verbatim, and 2 thematic analyses were performed. Results: Most of the participants thought that the prototype concept library would be a very helpful resource for conducting repeatable research, but they specified that many requirements are needed before its development. Although all the participants stated that they were aware of some existing concept libraries, most of them expressed negative perceptions about them. The participants mentioned several facilitators that would stimulate them to share their work and reuse the work of others, and they pointed out several barriers that could inhibit them from sharing their work and reusing the work of others. The participants suggested some developments that they would like to see to improve reproducible research output using routine data. Conclusions: The study indicated that most interviewees valued a concept library for phenotypes. However, only half of the participants felt that they would contribute by providing definitions for the concept library, and they reported many barriers regarding sharing their work on a publicly accessible platform. Analysis of interviews and the focus group discussion revealed that different stakeholders have different requirements, facilitators, barriers, and concerns about a prototype concept library. UR - https://humanfactors.jmir.org/2022/1/e31021 UR - http://dx.doi.org/10.2196/31021 UR - http://www.ncbi.nlm.nih.gov/pubmed/35289755 ID - info:doi/10.2196/31021 ER - TY - JOUR AU - Jung, Hyesil AU - Yoo, Sooyoung AU - Kim, Seok AU - Heo, Eunjeong AU - Kim, Borham AU - Lee, Ho-Young AU - Hwang, Hee PY - 2022/3/11 TI - Patient-Level Fall Risk Prediction Using the Observational Medical Outcomes Partnership?s Common Data Model: Pilot Feasibility Study JO - JMIR Med Inform SP - e35104 VL - 10 IS - 3 KW - common data model KW - accidental falls KW - Observational Medical Outcomes Partnership KW - nursing records KW - medical informatics KW - health data KW - electronic health record KW - data model KW - prediction model KW - risk prediction KW - fall risk N2 - Background: Falls in acute care settings threaten patients? safety. Researchers have been developing fall risk prediction models and exploring risk factors to provide evidence-based fall prevention practices; however, such efforts are hindered by insufficient samples, limited covariates, and a lack of standardized methodologies that aid study replication. Objective: The objectives of this study were to (1) convert fall-related electronic health record data into the standardized Observational Medical Outcome Partnership's (OMOP) common data model format and (2) develop models that predict fall risk during 2 time periods. Methods: As a pilot feasibility test, we converted fall-related electronic health record data (nursing notes, fall risk assessment sheet, patient acuity assessment sheet, and clinical observation sheet) into standardized OMOP common data model format using an extraction, transformation, and load process. We developed fall risk prediction models for 2 time periods (within 7 days of admission and during the entire hospital stay) using 2 algorithms (least absolute shrinkage and selection operator logistic regression and random forest). Results: In total, 6277 nursing statements, 747,049,486 clinical observation sheet records, 1,554,775 fall risk scores, and 5,685,011 patient acuity scores were converted into OMOP common data model format. All our models (area under the receiver operating characteristic curve 0.692-0.726) performed better than the Hendrich II Fall Risk Model. Patient acuity score, fall history, age ?60 years, movement disorder, and central nervous system agents were the most important predictors in the logistic regression models. Conclusions: To enhance model performance further, we are currently converting all nursing records into the OMOP common data model data format, which will then be included in the models. Thus, in the near future, the performance of fall risk prediction models could be improved through the application of abundant nursing records and external validation. UR - https://medinform.jmir.org/2022/3/e35104 UR - http://dx.doi.org/10.2196/35104 UR - http://www.ncbi.nlm.nih.gov/pubmed/35275076 ID - info:doi/10.2196/35104 ER - TY - JOUR AU - Wang, Liya AU - Qiu, Hang AU - Luo, Li AU - Zhou, Li PY - 2022/2/25 TI - Age- and Sex-Specific Differences in Multimorbidity Patterns and Temporal Trends on Assessing Hospital Discharge Records in Southwest China: Network-Based Study JO - J Med Internet Res SP - e27146 VL - 24 IS - 2 KW - multimorbidity pattern KW - temporal trend KW - network analysis KW - multimorbidity prevalence KW - administrative data KW - longitudinal study KW - regional research N2 - Background: Multimorbidity represents a global health challenge, which requires a more global understanding of multimorbidity patterns and trends. However, the majority of studies completed to date have often relied on self-reported conditions, and a simultaneous assessment of the entire spectrum of chronic disease co-occurrence, especially in developing regions, has not yet been performed. Objective: We attempted to provide a multidimensional approach to understand the full spectrum of chronic disease co-occurrence among general inpatients in southwest China, in order to investigate multimorbidity patterns and temporal trends, and assess their age and sex differences. Methods: We conducted a retrospective cohort analysis based on 8.8 million hospital discharge records of about 5.0 million individuals of all ages from 2015 to 2019 in a megacity in southwest China. We examined all chronic diagnoses using the ICD-10 (International Classification of Diseases, 10th revision) codes at 3 digits and focused on chronic diseases with ?1% prevalence for each of the age and sex strata, which resulted in a total of 149 and 145 chronic diseases in males and females, respectively. We constructed multimorbidity networks in the general population based on sex and age, and used the cosine index to measure the co-occurrence of chronic diseases. Then, we divided the networks into communities and assessed their temporal trends. Results: The results showed complex interactions among chronic diseases, with more intensive connections among males and inpatients ?40 years old. A total of 9 chronic diseases were simultaneously classified as central diseases, hubs, and bursts in the multimorbidity networks. Among them, 5 diseases were common to both males and females, including hypertension, chronic ischemic heart disease, cerebral infarction, other cerebrovascular diseases, and atherosclerosis. The earliest leaps (degree leaps ?6) appeared at a disorder of glycoprotein metabolism that happened at 25-29 years in males, about 15 years earlier than in females. The number of chronic diseases in the community increased over time, but the new entrants did not replace the root of the community. Conclusions: Our multimorbidity network analysis identified specific differences in the co-occurrence of chronic diagnoses by sex and age, which could help in the design of clinical interventions for inpatient multimorbidity. UR - https://www.jmir.org/2022/2/e27146 UR - http://dx.doi.org/10.2196/27146 UR - http://www.ncbi.nlm.nih.gov/pubmed/35212632 ID - info:doi/10.2196/27146 ER - TY - JOUR AU - Schwartz, L. Jessica AU - Tseng, Eva AU - Maruthur, M. Nisa AU - Rouhizadeh, Masoud PY - 2022/2/24 TI - Identification of Prediabetes Discussions in Unstructured Clinical Documentation: Validation of a Natural Language Processing Algorithm JO - JMIR Med Inform SP - e29803 VL - 10 IS - 2 KW - prediabetes KW - prediabetes discussions KW - prediabetes management KW - chronic disease management KW - physician-patient communication KW - natural language processing KW - machine learning N2 - Background: Prediabetes affects 1 in 3 US adults. Most are not receiving evidence-based interventions, so understanding how providers discuss prediabetes with patients will inform how to improve their care. Objective: This study aimed to develop a natural language processing (NLP) algorithm using machine learning techniques to identify discussions of prediabetes in narrative documentation. Methods: We developed and applied a keyword search strategy to identify discussions of prediabetes in clinical documentation for patients with prediabetes. We manually reviewed matching notes to determine which represented actual prediabetes discussions. We applied 7 machine learning models against our manual annotation. Results: Machine learning classifiers were able to achieve classification results that were close to human performance with up to 98% precision and recall to identify prediabetes discussions in clinical documentation. Conclusions: We demonstrated that prediabetes discussions can be accurately identified using an NLP algorithm. This approach can be used to understand and identify prediabetes management practices in primary care, thereby informing interventions to improve guideline-concordant care. UR - https://medinform.jmir.org/2022/2/e29803 UR - http://dx.doi.org/10.2196/29803 UR - http://www.ncbi.nlm.nih.gov/pubmed/35200154 ID - info:doi/10.2196/29803 ER - TY - JOUR AU - Siegersma, R. Klaske AU - Evers, Maxime AU - Bots, H. Sophie AU - Groepenhoff, Floor AU - Appelman, Yolande AU - Hofstra, Leonard AU - Tulevski, I. Igor AU - Somsen, Aernout G. AU - den Ruijter, M. Hester AU - Spruit, Marco AU - Onland-Moret, Charlotte N. PY - 2022/1/25 TI - Development of a Pipeline for Adverse Drug Reaction Identification in Clinical Notes: Word Embedding Models and String Matching JO - JMIR Med Inform SP - e31063 VL - 10 IS - 1 KW - adverse drug reactions KW - word embeddings KW - clinical notes N2 - Background: Knowledge about adverse drug reactions (ADRs) in the population is limited because of underreporting, which hampers surveillance and assessment of drug safety. Therefore, gathering accurate information that can be retrieved from clinical notes about the incidence of ADRs is of great relevance. However, manual labeling of these notes is time-consuming, and automatization can improve the use of free-text clinical notes for the identification of ADRs. Furthermore, tools for language processing in languages other than English are not widely available. Objective: The aim of this study is to design and evaluate a method for automatic extraction of medication and Adverse Drug Reaction Identification in Clinical Notes (ADRIN). Methods: Dutch free-text clinical notes (N=277,398) and medication registrations (N=499,435) from the Cardiology Centers of the Netherlands database were used. All clinical notes were used to develop word embedding models. Vector representations of word embedding models and string matching with a medical dictionary (Medical Dictionary for Regulatory Activities [MedDRA]) were used for identification of ADRs and medication in a test set of clinical notes that were manually labeled. Several settings, including search area and punctuation, could be adjusted in the prototype to evaluate the optimal version of the prototype. Results: The ADRIN method was evaluated using a test set of 988 clinical notes written on the stop date of a drug. Multiple versions of the prototype were evaluated for a variety of tasks. Binary classification of ADR presence achieved the highest accuracy of 0.84. Reduced search area and inclusion of punctuation improved performance, whereas incorporation of the MedDRA did not improve the performance of the pipeline. Conclusions: The ADRIN method and prototype are effective in recognizing ADRs in Dutch clinical notes from cardiac diagnostic screening centers. Surprisingly, incorporation of the MedDRA did not result in improved identification on top of word embedding models. The implementation of the ADRIN tool may help increase the identification of ADRs, resulting in better care and saving substantial health care costs. UR - https://medinform.jmir.org/2022/1/e31063 UR - http://dx.doi.org/10.2196/31063 UR - http://www.ncbi.nlm.nih.gov/pubmed/35076407 ID - info:doi/10.2196/31063 ER - TY - JOUR AU - Ahne, Adrian AU - Fagherazzi, Guy AU - Tannier, Xavier AU - Czernichow, Thomas AU - Orchard, Francisco PY - 2022/1/18 TI - Improving Diabetes-Related Biomedical Literature Exploration in the Clinical Decision-making Process via Interactive Classification and Topic Discovery: Methodology Development Study JO - J Med Internet Res SP - e27434 VL - 24 IS - 1 KW - evidence-based medicine KW - clinical decision making KW - clinical decision support KW - digital health KW - medical informatics KW - transparency KW - hierarchical clustering KW - active learning KW - classification KW - memory consumption KW - natural language processing N2 - Background: The amount of available textual health data such as scientific and biomedical literature is constantly growing and becoming more and more challenging for health professionals to properly summarize those data and practice evidence-based clinical decision making. Moreover, the exploration of unstructured health text data is challenging for professionals without computer science knowledge due to limited time, resources, and skills. Current tools to explore text data lack ease of use, require high computational efforts, and incorporate domain knowledge and focus on topics of interest with difficulty. Objective: We developed a methodology able to explore and target topics of interest via an interactive user interface for health professionals with limited computer science knowledge. We aim to reach near state-of-the-art performance while reducing memory consumption, increasing scalability, and minimizing user interaction effort to improve the clinical decision-making process. The performance was evaluated on diabetes-related abstracts from PubMed. Methods: The methodology consists of 4 parts: (1) a novel interpretable hierarchical clustering of documents where each node is defined by headwords (words that best represent the documents in the node), (2) an efficient classification system to target topics, (3) minimized user interaction effort through active learning, and (4) a visual user interface. We evaluated our approach on 50,911 diabetes-related abstracts providing a hierarchical Medical Subject Headings (MeSH) structure, a unique identifier for a topic. Hierarchical clustering performance was compared against the implementation in the machine learning library scikit-learn. On a subset of 2000 randomly chosen diabetes abstracts, our active learning strategy was compared against 3 other strategies: random selection of training instances, uncertainty sampling that chooses instances about which the model is most uncertain, and an expected gradient length strategy based on convolutional neural networks (CNNs). Results: For the hierarchical clustering performance, we achieved an F1 score of 0.73 compared to 0.76 achieved by scikit-learn. Concerning active learning performance, after 200 chosen training samples based on these strategies, the weighted F1 score of all MeSH codes resulted in a satisfying 0.62 F1 score using our approach, 0.61 using the uncertainty strategy, 0.63 using the CNN, and 0.45 using the random strategy. Moreover, our methodology showed a constant low memory use with increased number of documents. Conclusions: We proposed an easy-to-use tool for health professionals with limited computer science knowledge who combine their domain knowledge with topic exploration and target specific topics of interest while improving transparency. Furthermore, our approach is memory efficient and highly parallelizable, making it interesting for large Big Data sets. This approach can be used by health professionals to gain deep insights into biomedical literature to ultimately improve the evidence-based clinical decision making process. UR - https://www.jmir.org/2022/1/e27434 UR - http://dx.doi.org/10.2196/27434 UR - http://www.ncbi.nlm.nih.gov/pubmed/35040795 ID - info:doi/10.2196/27434 ER - TY - JOUR AU - Ulrich, Hannes AU - Kock-Schoppenhauer, Ann-Kristin AU - Deppenwiese, Noemi AU - Gött, Robert AU - Kern, Jori AU - Lablans, Martin AU - Majeed, W. Raphael AU - Stöhr, R. Mark AU - Stausberg, Jürgen AU - Varghese, Julian AU - Dugas, Martin AU - Ingenerf, Josef PY - 2022/1/11 TI - Understanding the Nature of Metadata: Systematic Review JO - J Med Internet Res SP - e25440 VL - 24 IS - 1 KW - metadata KW - metadata definition KW - systematic review KW - data integration KW - data identification KW - data classification N2 - Background: Metadata are created to describe the corresponding data in a detailed and unambiguous way and is used for various applications in different research areas, for example, data identification and classification. However, a clear definition of metadata is crucial for further use. Unfortunately, extensive experience with the processing and management of metadata has shown that the term ?metadata? and its use is not always unambiguous. Objective: This study aimed to understand the definition of metadata and the challenges resulting from metadata reuse. Methods: A systematic literature search was performed in this study following the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for reporting on systematic reviews. Five research questions were identified to streamline the review process, addressing metadata characteristics, metadata standards, use cases, and problems encountered. This review was preceded by a harmonization process to achieve a general understanding of the terms used. Results: The harmonization process resulted in a clear set of definitions for metadata processing focusing on data integration. The following literature review was conducted by 10 reviewers with different backgrounds and using the harmonized definitions. This study included 81 peer-reviewed papers from the last decade after applying various filtering steps to identify the most relevant papers. The 5 research questions could be answered, resulting in a broad overview of the standards, use cases, problems, and corresponding solutions for the application of metadata in different research areas. Conclusions: Metadata can be a powerful tool for identifying, describing, and processing information, but its meaningful creation is costly and challenging. This review process uncovered many standards, use cases, problems, and solutions for dealing with metadata. The presented harmonized definitions and the new schema have the potential to improve the classification and generation of metadata by creating a shared understanding of metadata and its context. UR - https://www.jmir.org/2022/1/e25440 UR - http://dx.doi.org/10.2196/25440 UR - http://www.ncbi.nlm.nih.gov/pubmed/35014967 ID - info:doi/10.2196/25440 ER - TY - JOUR AU - Vaidyam, Aditya AU - Halamka, John AU - Torous, John PY - 2022/1/7 TI - Enabling Research and Clinical Use of Patient-Generated Health Data (the mindLAMP Platform): Digital Phenotyping Study JO - JMIR Mhealth Uhealth SP - e30557 VL - 10 IS - 1 KW - digital phenotyping KW - mHealth KW - apps KW - FHIR KW - digital health KW - health data KW - patient-generated health data KW - mobile health KW - smartphones KW - wearables KW - mobile apps KW - mental health, mobile phone N2 - Background: There is a growing need for the integration of patient-generated health data (PGHD) into research and clinical care to enable personalized, preventive, and interactive care, but technical and organizational challenges, such as the lack of standards and easy-to-use tools, preclude the effective use of PGHD generated from consumer devices, such as smartphones and wearables. Objective: This study outlines how we used mobile apps and semantic web standards such as HTTP 2.0, Representational State Transfer, JSON (JavaScript Object Notation), JSON Schema, Transport Layer Security (version 1.3), Advanced Encryption Standard-256, OpenAPI, HTML5, and Vega, in conjunction with patient and provider feedback to completely update a previous version of mindLAMP. Methods: The Learn, Assess, Manage, and Prevent (LAMP) platform addresses the abovementioned challenges in enhancing clinical insight by supporting research, data analysis, and implementation efforts around PGHD as an open-source solution with freely accessible and shared code. Results: With a simplified programming interface and novel data representation that captures additional metadata, the LAMP platform enables interoperability with existing Fast Healthcare Interoperability Resources?based health care systems as well as consumer wearables and services such as Apple HealthKit and Google Fit. The companion Cortex data analysis and machine learning toolkit offer robust support for artificial intelligence, behavioral feature extraction, interactive visualizations, and high-performance data processing through parallelization and vectorization techniques. Conclusions: The LAMP platform incorporates feedback from patients and clinicians alongside a standards-based approach to address these needs and functions across a wide range of use cases through its customizable and flexible components. These range from simple survey-based research to international consortiums capturing multimodal data to simple delivery of mindfulness exercises through personalized, just-in-time adaptive interventions. UR - https://mhealth.jmir.org/2022/1/e30557 UR - http://dx.doi.org/10.2196/30557 UR - http://www.ncbi.nlm.nih.gov/pubmed/34994710 ID - info:doi/10.2196/30557 ER - TY - JOUR AU - Chen, Qingyu AU - Rankine, Alex AU - Peng, Yifan AU - Aghaarabi, Elaheh AU - Lu, Zhiyong PY - 2021/12/30 TI - Benchmarking Effectiveness and Efficiency of Deep Learning Models for Semantic Textual Similarity in the Clinical Domain: Validation Study JO - JMIR Med Inform SP - e27386 VL - 9 IS - 12 KW - semantic textual similarity KW - deep learning KW - biomedical and clinical text mining KW - word embeddings KW - sentence embeddings KW - transformers N2 - Background: Semantic textual similarity (STS) measures the degree of relatedness between sentence pairs. The Open Health Natural Language Processing (OHNLP) Consortium released an expertly annotated STS data set and called for the National Natural Language Processing Clinical Challenges. This work describes our entry, an ensemble model that leverages a range of deep learning (DL) models. Our team from the National Library of Medicine obtained a Pearson correlation of 0.8967 in an official test set during 2019 National Natural Language Processing Clinical Challenges/Open Health Natural Language Processing shared task and achieved a second rank. Objective: Although our models strongly correlate with manual annotations, annotator-level correlation was only moderate (weighted Cohen ?=0.60). We are cautious of the potential use of DL models in production systems and argue that it is more critical to evaluate the models in-depth, especially those with extremely high correlations. In this study, we benchmark the effectiveness and efficiency of top-ranked DL models. We quantify their robustness and inference times to validate their usefulness in real-time applications. Methods: We benchmarked five DL models, which are the top-ranked systems for STS tasks: Convolutional Neural Network, BioSentVec, BioBERT, BlueBERT, and ClinicalBERT. We evaluated a random forest model as an additional baseline. For each model, we repeated the experiment 10 times, using the official training and testing sets. We reported 95% CI of the Wilcoxon rank-sum test on the average Pearson correlation (official evaluation metric) and running time. We further evaluated Spearman correlation, R², and mean squared error as additional measures. Results: Using only the official training set, all models obtained highly effective results. BioSentVec and BioBERT achieved the highest average Pearson correlations (0.8497 and 0.8481, respectively). BioSentVec also had the highest results in 3 of 4 effectiveness measures, followed by BioBERT. However, their robustness to sentence pairs of different similarity levels varies significantly. A particular observation is that BERT models made the most errors (a mean squared error of over 2.5) on highly similar sentence pairs. They cannot capture highly similar sentence pairs effectively when they have different negation terms or word orders. In addition, time efficiency is dramatically different from the effectiveness results. On average, the BERT models were approximately 20 times and 50 times slower than the Convolutional Neural Network and BioSentVec models, respectively. This results in challenges for real-time applications. Conclusions: Despite the excitement of further improving Pearson correlations in this data set, our results highlight that evaluations of the effectiveness and efficiency of STS models are critical. In future, we suggest more evaluations on the generalization capability and user-level testing of the models. We call for community efforts to create more biomedical and clinical STS data sets from different perspectives to reflect the multifaceted notion of sentence-relatedness. UR - https://medinform.jmir.org/2021/12/e27386 UR - http://dx.doi.org/10.2196/27386 UR - http://www.ncbi.nlm.nih.gov/pubmed/34967748 ID - info:doi/10.2196/27386 ER - TY - JOUR AU - Chopard, Daphne AU - Treder, S. Matthias AU - Corcoran, Padraig AU - Ahmed, Nagheen AU - Johnson, Claire AU - Busse, Monica AU - Spasic, Irena PY - 2021/12/24 TI - Text Mining of Adverse Events in Clinical Trials: Deep Learning Approach JO - JMIR Med Inform SP - e28632 VL - 9 IS - 12 KW - natural language processing KW - deep learning KW - machine learning KW - classification N2 - Background: Pharmacovigilance and safety reporting, which involve processes for monitoring the use of medicines in clinical trials, play a critical role in the identification of previously unrecognized adverse events or changes in the patterns of adverse events. Objective: This study aims to demonstrate the feasibility of automating the coding of adverse events described in the narrative section of the serious adverse event report forms to enable statistical analysis of the aforementioned patterns. Methods: We used the Uni?ed Medical Language System (UMLS) as the coding scheme, which integrates 217 source vocabularies, thus enabling coding against other relevant terminologies such as the International Classification of Diseases?10th Revision, Medical Dictionary for Regulatory Activities, and Systematized Nomenclature of Medicine). We used MetaMap, a highly configurable dictionary lookup software, to identify the mentions of the UMLS concepts. We trained a binary classifier using Bidirectional Encoder Representations from Transformers (BERT), a transformer-based language model that captures contextual relationships, to differentiate between mentions of the UMLS concepts that represented adverse events and those that did not. Results: The model achieved a high F1 score of 0.8080, despite the class imbalance. This is 10.15 percent points lower than human-like performance but also 17.45 percent points higher than that of the baseline approach. Conclusions: These results confirmed that automated coding of adverse events described in the narrative section of serious adverse event reports is feasible. Once coded, adverse events can be statistically analyzed so that any correlations with the trialed medicines can be estimated in a timely fashion. UR - https://medinform.jmir.org/2021/12/e28632 UR - http://dx.doi.org/10.2196/28632 UR - http://www.ncbi.nlm.nih.gov/pubmed/34951601 ID - info:doi/10.2196/28632 ER - TY - JOUR AU - Paris, Nicolas AU - Lamer, Antoine AU - Parrot, Adrien PY - 2021/12/14 TI - Transformation and Evaluation of the MIMIC Database in the OMOP Common Data Model: Development and Usability Study JO - JMIR Med Inform SP - e30970 VL - 9 IS - 12 KW - data reuse KW - open data KW - OMOP KW - common data model KW - critical care KW - machine learning KW - big data KW - health informatics KW - health data KW - health database KW - electronic health records KW - open access database KW - digital health KW - intensive care KW - health care N2 - Background: In the era of big data, the intensive care unit (ICU) is likely to benefit from real-time computer analysis and modeling based on close patient monitoring and electronic health record data. The Medical Information Mart for Intensive Care (MIMIC) is the first open access database in the ICU domain. Many studies have shown that common data models (CDMs) improve database searching by allowing code, tools, and experience to be shared. The Observational Medical Outcomes Partnership (OMOP) CDM is spreading all over the world. Objective: The objective was to transform MIMIC into an OMOP database and to evaluate the benefits of this transformation for analysts. Methods: We transformed MIMIC (version 1.4.21) into OMOP format (version 5.3.3.1) through semantic and structural mapping. The structural mapping aimed at moving the MIMIC data into the right place in OMOP, with some data transformations. The mapping was divided into 3 phases: conception, implementation, and evaluation. The conceptual mapping aimed at aligning the MIMIC local terminologies to OMOP's standard ones. It consisted of 3 phases: integration, alignment, and evaluation. A documented, tested, versioned, exemplified, and open repository was set up to support the transformation and improvement of the MIMIC community's source code. The resulting data set was evaluated over a 48-hour datathon. Results: With an investment of 2 people for 500 hours, 64% of the data items of the 26 MIMIC tables were standardized into the OMOP CDM and 78% of the source concepts mapped to reference terminologies. The model proved its ability to support community contributions and was well received during the datathon, with 160 participants and 15,000 requests executed with a maximum duration of 1 minute. Conclusions: The resulting MIMIC-OMOP data set is the first MIMIC-OMOP data set available free of charge with real disidentified data ready for replicable intensive care research. This approach can be generalized to any medical field. UR - https://medinform.jmir.org/2021/12/e30970 UR - http://dx.doi.org/10.2196/30970 UR - http://www.ncbi.nlm.nih.gov/pubmed/34904958 ID - info:doi/10.2196/30970 ER - TY - JOUR AU - Bannay, Aurélie AU - Bories, Mathilde AU - Le Corre, Pascal AU - Riou, Christine AU - Lemordant, Pierre AU - Van Hille, Pascal AU - Chazard, Emmanuel AU - Dode, Xavier AU - Cuggia, Marc AU - Bouzillé, Guillaume PY - 2021/12/13 TI - Leveraging National Claims and Hospital Big Data: Cohort Study on a Statin-Drug Interaction Use Case JO - JMIR Med Inform SP - e29286 VL - 9 IS - 12 KW - drug interactions KW - statins KW - administrative claims KW - health care KW - big data KW - data linking KW - data warehousing N2 - Background: Linking different sources of medical data is a promising approach to analyze care trajectories. The aim of the INSHARE (Integrating and Sharing Health Big Data for Research) project was to provide the blueprint for a technological platform that facilitates integration, sharing, and reuse of data from 2 sources: the clinical data warehouse (CDW) of the Rennes academic hospital, called eHOP (entrepôt Hôpital), and a data set extracted from the French national claim data warehouse (Système National des Données de Santé [SNDS]). Objective: This study aims to demonstrate how the INSHARE platform can support big data analytic tasks in the health field using a pharmacovigilance use case based on statin consumption and statin-drug interactions. Methods: A Spark distributed cluster-computing framework was used for the record linkage procedure and all analyses. A semideterministic record linkage method based on the common variables between the chosen data sources was developed to identify all patients discharged after at least one hospital stay at the Rennes academic hospital between 2015 and 2017. The use-case study focused on a cohort of patients treated with statins prescribed by their general practitioner or during their hospital stay. Results: The whole process (record linkage procedure and use-case analyses) required 88 minutes. Of the 161,532 and 164,316 patients from the SNDS and eHOP CDW data sets, respectively, 159,495 patients were successfully linked (98.74% and 97.07% of patients from SNDS and eHOP CDW, respectively). Of the 16,806 patients with at least one statin delivery, 8293 patients started the consumption before and continued during the hospital stay, 6382 patients stopped statin consumption at hospital admission, and 2131 patients initiated statins in hospital. Statin-drug interactions occurred more frequently during hospitalization than in the community (3800/10,424, 36.45% and 3253/14,675, 22.17%, respectively; P<.001). Only 121 patients had the most severe level of statin-drug interaction. Hospital stay burden (length of stay and in-hospital mortality) was more severe in patients with statin-drug interactions during hospitalization. Conclusions: This study demonstrates the added value of combining and reusing clinical and claim data to provide large-scale measures of drug-drug interaction prevalence and care pathways outside hospitals. It builds a path to move the current health care system toward a Learning Health System using knowledge generated from research on real-world health data. UR - https://medinform.jmir.org/2021/12/e29286 UR - http://dx.doi.org/10.2196/29286 UR - http://www.ncbi.nlm.nih.gov/pubmed/34898457 ID - info:doi/10.2196/29286 ER - TY - JOUR AU - Chan, Erina AU - Small, S. Serena AU - Wickham, E. Maeve AU - Cheng, Vicki AU - Balka, Ellen AU - Hohl, M. Corinne PY - 2021/12/10 TI - The Utility of Different Data Standards to Document Adverse Drug Event Symptoms and Diagnoses: Mixed Methods Study JO - J Med Internet Res SP - e27188 VL - 23 IS - 12 KW - adverse drug events KW - health information technology KW - data standards N2 - Background: Existing systems to document adverse drug events often use free text data entry, which produces nonstandardized and unstructured data that are prone to misinterpretation. Standardized terminology may improve data quality; however, it is unclear which data standard is most appropriate for documenting adverse drug event symptoms and diagnoses. Objective: This study aims to compare the utility, strengths, and weaknesses of different data standards for documenting adverse drug event symptoms and diagnoses. Methods: We performed a mixed methods substudy of a multicenter retrospective chart review. We reviewed the research records of prospectively diagnosed adverse drug events at 5 Canadian hospitals. A total of 2 pharmacy research assistants independently entered the symptoms and diagnoses for the adverse drug events using four standards: Medical Dictionary for Regulatory Activities (MedDRA), Systematized Nomenclature of Medicine (SNOMED) Clinical Terms, SNOMED Adverse Reaction (SNOMED ADR), and International Classification of Diseases (ICD) 11th Revision. Disagreements between research assistants regarding the case-specific utility of data standards were discussed until a consensus was reached. We used consensus ratings to determine the proportion of adverse drug events covered by a data standard and coded and analyzed field notes from the consensus sessions. Results: We reviewed 573 adverse drug events and found that MedDRA and ICD-11 had excellent coverage of adverse drug event symptoms and diagnoses. MedDRA had the highest number of matches between the research assistants, whereas ICD-11 had the fewest. SNOMED ADR had the lowest proportion of adverse drug event coverage. The research assistants were most likely to encounter terminological challenges with SNOMED ADR and usability challenges with ICD-11, whereas least likely to encounter challenges with MedDRA. Conclusions: Usability, comprehensiveness, and accuracy are important features of data standards for documenting adverse drug event symptoms and diagnoses. On the basis of our results, we recommend the use of MedDRA. UR - https://www.jmir.org/2021/12/e27188 UR - http://dx.doi.org/10.2196/27188 UR - http://www.ncbi.nlm.nih.gov/pubmed/34890351 ID - info:doi/10.2196/27188 ER - TY - JOUR AU - Pan, Youcheng AU - Wang, Chenghao AU - Hu, Baotian AU - Xiang, Yang AU - Wang, Xiaolong AU - Chen, Qingcai AU - Chen, Junjie AU - Du, Jingcheng PY - 2021/12/8 TI - A BERT-Based Generation Model to Transform Medical Texts to SQL Queries for Electronic Medical Records: Model Development and Validation JO - JMIR Med Inform SP - e32698 VL - 9 IS - 12 KW - electronic medical record KW - text-to-SQL generation KW - BERT KW - grammar-based decoding KW - tree-structured intermediate representation N2 - Background: Electronic medical records (EMRs) are usually stored in relational databases that require SQL queries to retrieve information of interest. Effectively completing such queries can be a challenging task for medical experts due to the barriers in expertise. Existing text-to-SQL generation studies have not been fully embraced in the medical domain. Objective: The objective of this study was to propose a neural generation model that can jointly consider the characteristics of medical text and the SQL structure to automatically transform medical texts to SQL queries for EMRs. Methods: We proposed a medical text?to-SQL model (MedTS), which employed a pretrained Bidirectional Encoder Representations From Transformers model as the encoder and leveraged a grammar-based long short-term memory network as the decoder to predict the intermediate representation that can easily be transformed into the final SQL query. We adopted the syntax tree as the intermediate representation rather than directly regarding the SQL query as an ordinary word sequence, which is more in line with the tree-structure nature of SQL and can also effectively reduce the search space during generation. Experiments were conducted on the MIMICSQL dataset, and 5 competitor methods were compared. Results: Experimental results demonstrated that MedTS achieved the accuracy of 0.784 and 0.899 on the test set in terms of logic form and execution, respectively, which significantly outperformed the existing state-of-the-art methods. Further analyses proved that the performance on each component of the generated SQL was relatively balanced and offered substantial improvements. Conclusions: The proposed MedTS was effective and robust for improving the performance of medical text?to-SQL generation, indicating strong potential to be applied in the real medical scenario. UR - https://medinform.jmir.org/2021/12/e32698 UR - http://dx.doi.org/10.2196/32698 UR - http://www.ncbi.nlm.nih.gov/pubmed/34889749 ID - info:doi/10.2196/32698 ER - TY - JOUR AU - Stöhr, R. Mark AU - Günther, Andreas AU - Majeed, W. Raphael PY - 2021/11/29 TI - The Collaborative Metadata Repository (CoMetaR) Web App: Quantitative and Qualitative Usability Evaluation JO - JMIR Med Inform SP - e30308 VL - 9 IS - 11 KW - usability KW - metadata KW - data visualization KW - semantic web KW - data management KW - data warehousing KW - communication barriers KW - quality improvement KW - biological ontologies KW - data curation N2 - Background: In the field of medicine and medical informatics, the importance of comprehensive metadata has long been recognized, and the composition of metadata has become its own field of profession and research. To ensure sustainable and meaningful metadata are maintained, standards and guidelines such as the FAIR (Findability, Accessibility, Interoperability, Reusability) principles have been published. The compilation and maintenance of metadata is performed by field experts supported by metadata management apps. The usability of these apps, for example, in terms of ease of use, efficiency, and error tolerance, crucially determines their benefit to those interested in the data. Objective: This study aims to provide a metadata management app with high usability that assists scientists in compiling and using rich metadata. We aim to evaluate our recently developed interactive web app for our collaborative metadata repository (CoMetaR). This study reflects how real users perceive the app by assessing usability scores and explicit usability issues. Methods: We evaluated the CoMetaR web app by measuring the usability of 3 modules: core module, provenance module, and data integration module. We defined 10 tasks in which users must acquire information specific to their user role. The participants were asked to complete the tasks in a live web meeting. We used the System Usability Scale questionnaire to measure the usability of the app. For qualitative analysis, we applied a modified think aloud method with the following thematic analysis and categorization into the ISO 9241-110 usability categories. Results: A total of 12 individuals participated in the study. We found that over 97% (85/88) of all the tasks were completed successfully. We measured usability scores of 81, 81, and 72 for the 3 evaluated modules. The qualitative analysis resulted in 24 issues with the app. Conclusions: A usability score of 81 implies very good usability for the 2 modules, whereas a usability score of 72 still indicates acceptable usability for the third module. We identified 24 issues that serve as starting points for further development. Our method proved to be effective and efficient in terms of effort and outcome. It can be adapted to evaluate apps within the medical informatics field and potentially beyond. UR - https://medinform.jmir.org/2021/11/e30308 UR - http://dx.doi.org/10.2196/30308 UR - http://www.ncbi.nlm.nih.gov/pubmed/34847059 ID - info:doi/10.2196/30308 ER - TY - JOUR AU - Dey, Vishal AU - Krasniak, Peter AU - Nguyen, Minh AU - Lee, Clara AU - Ning, Xia PY - 2021/11/29 TI - A Pipeline to Understand Emerging Illness Via Social Media Data Analysis: Case Study on Breast Implant Illness JO - JMIR Med Inform SP - e29768 VL - 9 IS - 11 KW - breast implant illness KW - social media KW - natural language processing KW - topic modeling N2 - Background: A new illness can come to public attention through social media before it is medically defined, formally documented, or systematically studied. One example is a condition known as breast implant illness (BII), which has been extensively discussed on social media, although it is vaguely defined in the medical literature. Objective: The objective of this study is to construct a data analysis pipeline to understand emerging illnesses using social media data and to apply the pipeline to understand the key attributes of BII. Methods: We constructed a pipeline of social media data analysis using natural language processing and topic modeling. Mentions related to signs, symptoms, diseases, disorders, and medical procedures were extracted from social media data using the clinical Text Analysis and Knowledge Extraction System. We mapped the mentions to standard medical concepts and then summarized these mapped concepts as topics using latent Dirichlet allocation. Finally, we applied this pipeline to understand BII from several BII-dedicated social media sites. Results: Our pipeline identified topics related to toxicity, cancer, and mental health issues that were highly associated with BII. Our pipeline also showed that cancers, autoimmune disorders, and mental health problems were emerging concerns associated with breast implants, based on social media discussions. Furthermore, the pipeline identified mentions such as rupture, infection, pain, and fatigue as common self-reported issues among the public, as well as concerns about toxicity from silicone implants. Conclusions: Our study could inspire future studies on the suggested symptoms and factors of BII. Our study provides the first analysis and derived knowledge of BII from social media using natural language processing techniques and demonstrates the potential of using social media information to better understand similar emerging illnesses. UR - https://medinform.jmir.org/2021/11/e29768 UR - http://dx.doi.org/10.2196/29768 UR - http://www.ncbi.nlm.nih.gov/pubmed/34847064 ID - info:doi/10.2196/29768 ER - TY - JOUR AU - Abdelkader, Wael AU - Navarro, Tamara AU - Parrish, Rick AU - Cotoi, Chris AU - Germini, Federico AU - Linkins, Lori-Ann AU - Iorio, Alfonso AU - Haynes, Brian R. AU - Ananiadou, Sophia AU - Chu, Lingyang AU - Lokker, Cynthia PY - 2021/11/29 TI - A Deep Learning Approach to Refine the Identification of High-Quality Clinical Research Articles From the Biomedical Literature: Protocol for Algorithm Development and Validation JO - JMIR Res Protoc SP - e29398 VL - 10 IS - 11 KW - bioinformatics KW - machine learning KW - evidence-based medicine KW - literature retrieval KW - medical informatics KW - natural language processing KW - NLP KW - biomedical KW - literature KW - literature surveillance KW - model development N2 - Background: A barrier to practicing evidence-based medicine is the rapidly increasing body of biomedical literature. Use of method terms to limit the search can help reduce the burden of screening articles for clinical relevance; however, such terms are limited by their partial dependence on indexing terms and usually produce low precision, especially when high sensitivity is required. Machine learning has been applied to the identification of high-quality literature with the potential to achieve high precision without sacrificing sensitivity. The use of artificial intelligence has shown promise to improve the efficiency of identifying sound evidence. Objective: The primary objective of this research is to derive and validate deep learning machine models using iterations of Bidirectional Encoder Representations from Transformers (BERT) to retrieve high-quality, high-relevance evidence for clinical consideration from the biomedical literature. Methods: Using the HuggingFace Transformers library, we will experiment with variations of BERT models, including BERT, BioBERT, BlueBERT, and PubMedBERT, to determine which have the best performance in article identification based on quality criteria. Our experiments will utilize a large data set of over 150,000 PubMed citations from 2012 to 2020 that have been manually labeled based on their methodological rigor for clinical use. We will evaluate and report on the performance of the classifiers in categorizing articles based on their likelihood of meeting quality criteria. We will report fine-tuning hyperparameters for each model, as well as their performance metrics, including recall (sensitivity), specificity, precision, accuracy, F-score, the number of articles that need to be read before finding one that is positive (meets criteria), and classification probability scores. Results: Initial model development is underway, with further development planned for early 2022. Performance testing is expected to star in February 2022. Results will be published in 2022. Conclusions: The experiments will aim to improve the precision of retrieving high-quality articles by applying a machine learning classifier to PubMed searching. International Registered Report Identifier (IRRID): DERR1-10.2196/29398 UR - https://www.researchprotocols.org/2021/11/e29398 UR - http://dx.doi.org/10.2196/29398 UR - http://www.ncbi.nlm.nih.gov/pubmed/34847061 ID - info:doi/10.2196/29398 ER - TY - JOUR AU - Chang, David AU - Lin, Eric AU - Brandt, Cynthia AU - Taylor, Andrew Richard PY - 2021/11/26 TI - Incorporating Domain Knowledge Into Language Models by Using Graph Convolutional Networks for Assessing Semantic Textual Similarity: Model Development and Performance Comparison JO - JMIR Med Inform SP - e23101 VL - 9 IS - 11 KW - natural language processing KW - graph neural networks KW - National NLP Clinical Challenges KW - bidirectional encoder representation from transformers N2 - Background: Although electronic health record systems have facilitated clinical documentation in health care, they have also introduced new challenges, such as the proliferation of redundant information through the use of copy and paste commands or templates. One approach to trimming down bloated clinical documentation and improving clinical summarization is to identify highly similar text snippets with the goal of removing such text. Objective: We developed a natural language processing system for the task of assessing clinical semantic textual similarity. The system assigns scores to pairs of clinical text snippets based on their clinical semantic similarity. Methods: We leveraged recent advances in natural language processing and graph representation learning to create a model that combines linguistic and domain knowledge information from the MedSTS data set to assess clinical semantic textual similarity. We used bidirectional encoder representation from transformers (BERT)?based models as text encoders for the sentence pairs in the data set and graph convolutional networks (GCNs) as graph encoders for corresponding concept graphs that were constructed based on the sentences. We also explored techniques, including data augmentation, ensembling, and knowledge distillation, to improve the model?s performance, as measured by the Pearson correlation coefficient (r). Results: Fine-tuning the BERT_base and ClinicalBERT models on the MedSTS data set provided a strong baseline (Pearson correlation coefficients: 0.842 and 0.848, respectively) compared to those of the previous year?s submissions. Our data augmentation techniques yielded moderate gains in performance, and adding a GCN-based graph encoder to incorporate the concept graphs also boosted performance, especially when the node features were initialized with pretrained knowledge graph embeddings of the concepts (r=0.868). As expected, ensembling improved performance, and performing multisource ensembling by using different language model variants, conducting knowledge distillation with the multisource ensemble model, and taking a final ensemble of the distilled models further improved the system?s performance (Pearson correlation coefficients: 0.875, 0.878, and 0.882, respectively). Conclusions: This study presents a system for the MedSTS clinical semantic textual similarity benchmark task, which was created by combining BERT-based text encoders and GCN-based graph encoders in order to incorporate domain knowledge into the natural language processing pipeline. We also experimented with other techniques involving data augmentation, pretrained concept embeddings, ensembling, and knowledge distillation to further increase our system?s performance. Although the task and its benchmark data set are in the early stages of development, this study, as well as the results of the competition, demonstrates the potential of modern language model?based systems to detect redundant information in clinical notes. UR - https://medinform.jmir.org/2021/11/e23101 UR - http://dx.doi.org/10.2196/23101 UR - http://www.ncbi.nlm.nih.gov/pubmed/34842531 ID - info:doi/10.2196/23101 ER - TY - JOUR AU - Pankhurst, Tanya AU - Evison, Felicity AU - Atia, Jolene AU - Gallier, Suzy AU - Coleman, Jamie AU - Ball, Simon AU - McKee, Deborah AU - Ryan, Steven AU - Black, Ruth PY - 2021/11/23 TI - Introduction of Systematized Nomenclature of Medicine?Clinical Terms Coding Into an Electronic Health Record and Evaluation of its Impact: Qualitative and Quantitative Study JO - JMIR Med Inform SP - e29532 VL - 9 IS - 11 KW - coding standards KW - clinical decision support KW - Clinician led design KW - clinician reported experience KW - clinical usability KW - data sharing KW - diagnoses KW - electronic health records KW - electronic health record standards KW - health data exchange KW - health data research KW - International Classification of Diseases version 10 (ICD-10) KW - National Health Service Blueprint KW - patient diagnoses KW - population health KW - problem list KW - research KW - Systematized Nomenclature Of Medicine?Clinical Terms (SNOMED-CT) KW - use of electronic health data KW - user-led design N2 - Background: This study describes the conversion within an existing electronic health record (EHR) from the International Classification of Diseases, Tenth Revision coding system to the SNOMED-CT (Systematized Nomenclature of Medicine?Clinical Terms) for the collection of patient histories and diagnoses. The setting is a large acute hospital that is designing and building its own EHR. Well-designed EHRs create opportunities for continuous data collection, which can be used in clinical decision support rules to drive patient safety. Collected data can be exchanged across health care systems to support patients in all health care settings. Data can be used for research to prevent diseases and protect future populations. Objective: The aim of this study was to migrate a current EHR, with all relevant patient data, to the SNOMED-CT coding system to optimize clinical use and clinical decision support, facilitate data sharing across organizational boundaries for national programs, and enable remodeling of medical pathways. Methods: The study used qualitative and quantitative data to understand the successes and gaps in the project, clinician attitudes toward the new tool, and the future use of the tool. Results: The new coding system (tool) was well received and immediately widely used in all specialties. This resulted in increased, accurate, and clinically relevant data collection. Clinicians appreciated the increased depth and detail of the new coding, welcomed the potential for both data sharing and research, and provided extensive feedback for further development. Conclusions: Successful implementation of the new system aligned the University Hospitals Birmingham NHS Foundation Trust with national strategy and can be used as a blueprint for similar projects in other health care settings. UR - https://medinform.jmir.org/2021/11/e29532 UR - http://dx.doi.org/10.2196/29532 UR - http://www.ncbi.nlm.nih.gov/pubmed/34817387 ID - info:doi/10.2196/29532 ER - TY - JOUR AU - Lu, Zhaohua AU - Sim, Jin-ah AU - Wang, X. Jade AU - Forrest, B. Christopher AU - Krull, R. Kevin AU - Srivastava, Deokumar AU - Hudson, M. Melissa AU - Robison, L. Leslie AU - Baker, N. Justin AU - Huang, I-Chan PY - 2021/11/3 TI - Natural Language Processing and Machine Learning Methods to Characterize Unstructured Patient-Reported Outcomes: Validation Study JO - J Med Internet Res SP - e26777 VL - 23 IS - 11 KW - natural language processing KW - machine learning KW - PROs KW - pediatric oncology N2 - Background: Assessing patient-reported outcomes (PROs) through interviews or conversations during clinical encounters provides insightful information about survivorship. Objective: This study aims to test the validity of natural language processing (NLP) and machine learning (ML) algorithms in identifying different attributes of pain interference and fatigue symptoms experienced by child and adolescent survivors of cancer versus the judgment by PRO content experts as the gold standard to validate NLP/ML algorithms. Methods: This cross-sectional study focused on child and adolescent survivors of cancer, aged 8 to 17 years, and caregivers, from whom 391 meaning units in the pain interference domain and 423 in the fatigue domain were generated for analyses. Data were collected from the After Completion of Therapy Clinic at St. Jude Children?s Research Hospital. Experienced pain interference and fatigue symptoms were reported through in-depth interviews. After verbatim transcription, analyzable sentences (ie, meaning units) were semantically labeled by 2 content experts for each attribute (physical, cognitive, social, or unclassified). Two NLP/ML methods were used to extract and validate the semantic features: bidirectional encoder representations from transformers (BERT) and Word2vec plus one of the ML methods, the support vector machine or extreme gradient boosting. Receiver operating characteristic and precision-recall curves were used to evaluate the accuracy and validity of the NLP/ML methods. Results: Compared with Word2vec/support vector machine and Word2vec/extreme gradient boosting, BERT demonstrated higher accuracy in both symptom domains, with 0.931 (95% CI 0.905-0.957) and 0.916 (95% CI 0.887-0.941) for problems with cognitive and social attributes on pain interference, respectively, and 0.929 (95% CI 0.903-0.953) and 0.917 (95% CI 0.891-0.943) for problems with cognitive and social attributes on fatigue, respectively. In addition, BERT yielded superior areas under the receiver operating characteristic curve for cognitive attributes on pain interference and fatigue domains (0.923, 95% CI 0.879-0.997; 0.948, 95% CI 0.922-0.979) and superior areas under the precision-recall curve for cognitive attributes on pain interference and fatigue domains (0.818, 95% CI 0.735-0.917; 0.855, 95% CI 0.791-0.930). Conclusions: The BERT method performed better than the other methods. As an alternative to using standard PRO surveys, collecting unstructured PROs via interviews or conversations during clinical encounters and applying NLP/ML methods can facilitate PRO assessment in child and adolescent cancer survivors. UR - https://www.jmir.org/2021/11/e26777 UR - http://dx.doi.org/10.2196/26777 UR - http://www.ncbi.nlm.nih.gov/pubmed/34730546 ID - info:doi/10.2196/26777 ER - TY - JOUR AU - Zanotto, Stella Bruna AU - Beck da Silva Etges, Paula Ana AU - dal Bosco, Avner AU - Cortes, Gabriel Eduardo AU - Ruschel, Renata AU - De Souza, Claudia Ana AU - Andrade, V. Claudio M. AU - Viegas, Felipe AU - Canuto, Sergio AU - Luiz, Washington AU - Ouriques Martins, Sheila AU - Vieira, Renata AU - Polanczyk, Carisi AU - André Gonçalves, Marcos PY - 2021/11/1 TI - Stroke Outcome Measurements From Electronic Medical Records: Cross-sectional Study on the Effectiveness of Neural and Nonneural Classifiers JO - JMIR Med Inform SP - e29120 VL - 9 IS - 11 KW - natural language processing KW - stroke KW - outcomes KW - electronic medical records KW - EHR KW - electronic health records KW - text processing KW - data mining KW - text classification KW - patient outcomes N2 - Background: With the rapid adoption of electronic medical records (EMRs), there is an ever-increasing opportunity to collect data and extract knowledge from EMRs to support patient-centered stroke management. Objective: This study aims to compare the effectiveness of state-of-the-art automatic text classification methods in classifying data to support the prediction of clinical patient outcomes and the extraction of patient characteristics from EMRs. Methods: Our study addressed the computational problems of information extraction and automatic text classification. We identified essential tasks to be considered in an ischemic stroke value-based program. The 30 selected tasks were classified (manually labeled by specialists) according to the following value agenda: tier 1 (achieved health care status), tier 2 (recovery process), care related (clinical management and risk scores), and baseline characteristics. The analyzed data set was retrospectively extracted from the EMRs of patients with stroke from a private Brazilian hospital between 2018 and 2019. A total of 44,206 sentences from free-text medical records in Portuguese were used to train and develop 10 supervised computational machine learning methods, including state-of-the-art neural and nonneural methods, along with ontological rules. As an experimental protocol, we used a 5-fold cross-validation procedure repeated 6 times, along with subject-wise sampling. A heatmap was used to display comparative result analyses according to the best algorithmic effectiveness (F1 score), supported by statistical significance tests. A feature importance analysis was conducted to provide insights into the results. Results: The top-performing models were support vector machines trained with lexical and semantic textual features, showing the importance of dealing with noise in EMR textual representations. The support vector machine models produced statistically superior results in 71% (17/24) of tasks, with an F1 score >80% regarding care-related tasks (patient treatment location, fall risk, thrombolytic therapy, and pressure ulcer risk), the process of recovery (ability to feed orally or ambulate and communicate), health care status achieved (mortality), and baseline characteristics (diabetes, obesity, dyslipidemia, and smoking status). Neural methods were largely outperformed by more traditional nonneural methods, given the characteristics of the data set. Ontological rules were also effective in tasks such as baseline characteristics (alcoholism, atrial fibrillation, and coronary artery disease) and the Rankin scale. The complementarity in effectiveness among models suggests that a combination of models could enhance the results and cover more tasks in the future. Conclusions: Advances in information technology capacity are essential for scalability and agility in measuring health status outcomes. This study allowed us to measure effectiveness and identify opportunities for automating the classification of outcomes of specific tasks related to clinical conditions of stroke victims, and thus ultimately assess the possibility of proactively using these machine learning techniques in real-world situations. UR - https://medinform.jmir.org/2021/11/e29120 UR - http://dx.doi.org/10.2196/29120 UR - http://www.ncbi.nlm.nih.gov/pubmed/34723829 ID - info:doi/10.2196/29120 ER - TY - JOUR AU - Lamer, Antoine AU - Abou-Arab, Osama AU - Bourgeois, Alexandre AU - Parrot, Adrien AU - Popoff, Benjamin AU - Beuscart, Jean-Baptiste AU - Tavernier, Benoît AU - Moussa, Djahoum Mouhamed PY - 2021/10/29 TI - Transforming Anesthesia Data Into the Observational Medical Outcomes Partnership Common Data Model: Development and Usability Study JO - J Med Internet Res SP - e29259 VL - 23 IS - 10 KW - data reuse KW - common data model KW - Observational Medical Outcomes Partnership KW - anesthesia KW - data warehouse KW - reproducible research N2 - Background: Electronic health records (EHRs, such as those created by an anesthesia management system) generate a large amount of data that can notably be reused for clinical audits and scientific research. The sharing of these data and tools is generally affected by the lack of system interoperability. To overcome these issues, Observational Health Data Sciences and Informatics (OHDSI) developed the Observational Medical Outcomes Partnership (OMOP) common data model (CDM) to standardize EHR data and promote large-scale observational and longitudinal research. Anesthesia data have not previously been mapped into the OMOP CDM. Objective: The primary objective was to transform anesthesia data into the OMOP CDM. The secondary objective was to provide vocabularies, queries, and dashboards that might promote the exploitation and sharing of anesthesia data through the CDM. Methods: Using our local anesthesia data warehouse, a group of 5 experts from 5 different medical centers identified local concepts related to anesthesia. The concepts were then matched with standard concepts in the OHDSI vocabularies. We performed structural mapping between the design of our local anesthesia data warehouse and the OMOP CDM tables and fields. To validate the implementation of anesthesia data into the OMOP CDM, we developed a set of queries and dashboards. Results: We identified 522 concepts related to anesthesia care. They were classified as demographics, units, measurements, operating room steps, drugs, periods of interest, and features. After semantic mapping, 353 (67.7%) of these anesthesia concepts were mapped to OHDSI concepts. Further, 169 (32.3%) concepts related to periods and features were added to the OHDSI vocabularies. Then, 8 OMOP CDM tables were implemented with anesthesia data and 2 new tables (EPISODE and FEATURE) were added to store secondarily computed data. We integrated data from 5,72,609 operations and provided the code for a set of 8 queries and 4 dashboards related to anesthesia care. Conclusions: Generic data concerning demographics, drugs, units, measurements, and operating room steps were already available in OHDSI vocabularies. However, most of the intraoperative concepts (the duration of specific steps, an episode of hypotension, etc) were not present in OHDSI vocabularies. The OMOP mapping provided here enables anesthesia data reuse. UR - https://www.jmir.org/2021/10/e29259 UR - http://dx.doi.org/10.2196/29259 UR - http://www.ncbi.nlm.nih.gov/pubmed/34714250 ID - info:doi/10.2196/29259 ER - TY - JOUR AU - Tsuji, Shintaro AU - Wen, Andrew AU - Takahashi, Naoki AU - Zhang, Hongjian AU - Ogasawara, Katsuhiko AU - Jiang, Gouqian PY - 2021/10/29 TI - Developing a RadLex-Based Named Entity Recognition Tool for Mining Textual Radiology Reports: Development and Performance Evaluation Study JO - J Med Internet Res SP - e25378 VL - 23 IS - 10 KW - named entity recognition (NER) KW - natural language processing (NLP) KW - RadLex KW - ontology KW - stem term N2 - Background: Named entity recognition (NER) plays an important role in extracting the features of descriptions such as the name and location of a disease for mining free-text radiology reports. However, the performance of existing NER tools is limited because the number of entities that can be extracted depends on the dictionary lookup. In particular, the recognition of compound terms is very complicated because of the variety of patterns. Objective: The aim of this study is to develop and evaluate an NER tool concerned with compound terms using RadLex for mining free-text radiology reports. Methods: We leveraged the clinical Text Analysis and Knowledge Extraction System (cTAKES) to develop customized pipelines using both RadLex and SentiWordNet (a general purpose dictionary). We manually annotated 400 radiology reports for compound terms in noun phrases and used them as the gold standard for performance evaluation (precision, recall, and F-measure). In addition, we created a compound terms?enhanced dictionary (CtED) by analyzing false negatives and false positives and applied it to another 100 radiology reports for validation. We also evaluated the stem terms of compound terms by defining two measures: occurrence ratio (OR) and matching ratio (MR). Results: The F-measure of cTAKES+RadLex+general purpose dictionary was 30.9% (precision 73.3% and recall 19.6%) and that of the combined CtED was 63.1% (precision 82.8% and recall 51%). The OR indicated that the stem terms of effusion, node, tube, and disease were used frequently, but it still lacks capturing compound terms. The MR showed that 71.85% (9411/13,098) of the stem terms matched with that of the ontologies, and RadLex improved approximately 22% of the MR from the cTAKES default dictionary. The OR and MR revealed that the characteristics of stem terms would have the potential to help generate synonymous phrases using the ontologies. Conclusions: We developed a RadLex-based customized pipeline for parsing radiology reports and demonstrated that CtED and stem term analysis has the potential to improve dictionary-based NER performance with regard to expanding vocabularies. UR - https://www.jmir.org/2021/10/e25378 UR - http://dx.doi.org/10.2196/25378 UR - http://www.ncbi.nlm.nih.gov/pubmed/34714247 ID - info:doi/10.2196/25378 ER - TY - JOUR AU - Berenspöhler, Sarah AU - Minnerup, Jens AU - Dugas, Martin AU - Varghese, Julian PY - 2021/10/12 TI - Common Data Elements for Meaningful Stroke Documentation in Routine Care and Clinical Research: Retrospective Data Analysis JO - JMIR Med Inform SP - e27396 VL - 9 IS - 10 KW - common data elements KW - stroke KW - documentation N2 - Background: Medical information management for stroke patients is currently a very time-consuming endeavor. There are clear guidelines and procedures to treat patients having acute stroke, but it is not known how well these established practices are reflected in patient documentation. Objective: This study compares a variety of documentation processes regarding stroke. The main objective of this work is to provide an overview of the most commonly occurring medical concepts in stroke documentation and identify overlaps between different documentation contexts to allow for the definition of a core data set that could be used in potential data interfaces. Methods: Medical source documentation forms from different documentation contexts, including hospitals, clinical trials, registries, and international standards, regarding stroke treatment followed by rehabilitation were digitized in the operational data model. Each source data element was semantically annotated using the Unified Medical Language System. The concept codes were analyzed for semantic overlaps. A concept was considered common if it appeared in at least two documentation contexts. The resulting common concepts were extended with implementation details, including data types and permissible values based on frequent patterns of source data elements, using an established expert-based and semiautomatic approach. Results: In total, 3287 data elements were identified, and 1051 of these emerged as unique medical concepts. The 100 most frequent medical concepts cover 9.51% (100/1051) of all concept occurrences in stroke documentation, and the 50 most frequent concepts cover 4.75% (50/1051). A list of common data elements was implemented in different standardized machine-readable formats on a public metadata repository for interoperable reuse. Conclusions: Standardization of medical documentation is a prerequisite for data exchange as well as the transferability and reuse of data. In the long run, standardization would save time and money and extend the capabilities for which such data could be used. In the context of this work, a lack of standardization was observed regarding current information management. Free-form text fields and intricate questions complicate automated data access and transfer between institutions. This work also revealed the potential of a unified documentation process as a core data set of the 50 most frequent common data elements, accounting for 34% of the documentation in medical information management. Such a data set offers a starting point for standardized and interoperable data collection in routine care, quality management, and clinical research. UR - https://medinform.jmir.org/2021/10/e27396 UR - http://dx.doi.org/10.2196/27396 UR - http://www.ncbi.nlm.nih.gov/pubmed/34636733 ID - info:doi/10.2196/27396 ER - TY - JOUR AU - Hüsers, Jens AU - Przysucha, Mareike AU - Esdar, Moritz AU - John, Malte Swen AU - Hübner, Hertha Ursula PY - 2021/10/6 TI - Expressiveness of an International Semantic Standard for Wound Care: Mapping a Standardized Item Set for Leg Ulcers to the Systematized Nomenclature of Medicine?Clinical Terms JO - JMIR Med Inform SP - e31980 VL - 9 IS - 10 KW - wound care KW - chronic wound KW - chronic leg ulcer KW - SNOMED CT KW - health information exchange KW - semantic interoperability KW - terminology mapping N2 - Background: Chronic health conditions are on the rise and are putting high economic pressure on health systems, as they require well-coordinated prevention and treatment. Among chronic conditions, chronic wounds such as cardiovascular leg ulcers have a high prevalence. Their treatment is highly interdisciplinary and regularly spans multiple care settings and organizations; this places particularly high demands on interoperable information exchange that can be achieved using international semantic standards, such as Systematized Nomenclature of Medicine?Clinical Terms (SNOMED CT). Objective: This study aims to investigate the expressiveness of SNOMED CT in the domain of wound care, and thereby its clinical usefulness and the potential need for extensions. Methods: A clinically consented and profession-independent wound care item set, the German National Consensus for the Documentation of Leg Wounds (NKDUC), was mapped onto the precoordinated concepts of the international reference terminology SNOMED CT. Before the mapping took place, the NKDUC was transformed into an information model that served to systematically identify relevant items. The mapping process was carried out in accordance with the ISO/TR 12300 formalism. As a result, the reliability, equivalence, and coverage rate were determined for all NKDUC items and sections. Results: The developed information model revealed 268 items to be mapped. Conducted by 3 health care professionals, the mapping resulted in moderate reliability (?=0.512). Regarding the two best equivalence categories (symmetrical equivalence of meaning), the coverage rate of SNOMED CT was 67.2% (180/268) overall and 64.3% (108/168) specifically for wounds. The sections general medical condition (55/66, 83%), wound assessment (18/24, 75%), and wound status (37/57, 65%), showed higher coverage rates compared with the sections therapy (45/73, 62%), wound diagnostics (8/14, 57%), and patient demographics (17/34, 50%). Conclusions: The results yielded acceptable reliability values for the mapping procedure. The overall coverage rate shows that two-thirds of the items could be mapped symmetrically, which is a substantial portion of the source item set. Some wound care sections, such as general medical conditions and wound assessment, were covered better than other sections (wound status, diagnostics, and therapy). These deficiencies can be mitigated either by postcoordination or by the inclusion of new concepts in SNOMED CT. This study contributes to pushing interoperability in the domain of wound care, thereby responding to the high demand for information exchange in this field. Overall, this study adds another puzzle piece to the general knowledge about SNOMED CT in terms of its clinical usefulness and its need for further extensions. UR - https://medinform.jmir.org/2021/10/e31980 UR - http://dx.doi.org/10.2196/31980 UR - http://www.ncbi.nlm.nih.gov/pubmed/34428171 ID - info:doi/10.2196/31980 ER - TY - JOUR AU - Glöggler, Michael AU - Ammenwerth, Elske PY - 2021/10/5 TI - Improvement and Evaluation of the TOPCOP Taxonomy of Patient Portals: Taxonomy-Evaluation-Delphi (TED) Approach JO - J Med Internet Res SP - e30701 VL - 23 IS - 10 KW - taxonomy KW - classification system KW - patient portal KW - EHR portal KW - online EHR access KW - evaluation KW - Delphi study KW - electronic health records KW - digital health KW - health information KW - information management KW - user perspectives N2 - Background: Patient portals have been introduced in many countries over the last 10 years, but many health information managers still feel they have too little knowledge of patient portals. A taxonomy can help them to better compare and select portals. This has led us to develop the TOPCOP taxonomy for classifying and comparing patient portals. However, the taxonomy has not been evaluated by users. Objective: This study aimed to evaluate the taxonomy?s usefulness to support health information managers in comparing, classifying, defining a requirement profile for, and selecting patient portals and to improve the taxonomy where needed. Methods: We used a modified Delphi approach. We sampled a heterogeneous panel of 13 health information managers from 3 countries using the criterion sampling strategy. We conducted 4 anonymous survey rounds with qualitative and quantitative questions. In round 1, the panelists assessed the appropriateness of each dimension, and we collected new ideas to improve the dimensions. In rounds 2 and 3, the panelists iteratively evaluated the taxonomy that was revised based on round 1. In round 4, the panelists assessed the need for a taxonomy and the appropriateness of patient engagement as a distinguishing concept. Then, they compared 2 real portals with the final taxonomy and evaluated its usefulness for comparing portals, creating an initial requirement profile, and selecting patient portals. To determine group consensus, we applied the RAND/UCLA Appropriateness Method. Results: The final taxonomy consists of 25 dimensions with 65 characteristics. Five new dimensions were added to the original taxonomy, with 8 characteristics added to already existing dimensions. Group consensus was achieved on the need for such a taxonomy to compare portals, on patient engagement as an appropriate distinguishing concept, and on the comprehensibility of the taxonomy?s form. Further, consensus was achieved on the taxonomy?s usefulness for classifying and comparing portals, assisting users in better understanding portals, creating a requirement profile, and selecting portals. This allowed us to test the usefulness of the final taxonomy with the intended users. Conclusions: The TOPCOP taxonomy aims to support health information managers in comparing and selecting patient portals. By providing a standardized terminology to describe various aspects of patient portals independent of clinical setting or country, the taxonomy will also be useful for advancing research and evaluation of patient portals. UR - https://www.jmir.org/2021/10/e30701 UR - http://dx.doi.org/10.2196/30701 UR - http://www.ncbi.nlm.nih.gov/pubmed/34403354 ID - info:doi/10.2196/30701 ER - TY - JOUR AU - Teodoro, Douglas AU - Ferdowsi, Sohrab AU - Borissov, Nikolay AU - Kashani, Elham AU - Vicente Alvarez, David AU - Copara, Jenny AU - Gouareb, Racha AU - Naderi, Nona AU - Amini, Poorya PY - 2021/9/17 TI - Information Retrieval in an Infodemic: The Case of COVID-19 Publications JO - J Med Internet Res SP - e30161 VL - 23 IS - 9 KW - information retrieval KW - multistage retrieval KW - neural search KW - deep learning KW - COVID-19 KW - coronavirus KW - infodemic KW - infodemiology KW - literature KW - online information N2 - Background: The COVID-19 global health crisis has led to an exponential surge in published scientific literature. In an attempt to tackle the pandemic, extremely large COVID-19?related corpora are being created, sometimes with inaccurate information, which is no longer at scale of human analyses. Objective: In the context of searching for scientific evidence in the deluge of COVID-19?related literature, we present an information retrieval methodology for effective identification of relevant sources to answer biomedical queries posed using natural language. Methods: Our multistage retrieval methodology combines probabilistic weighting models and reranking algorithms based on deep neural architectures to boost the ranking of relevant documents. Similarity of COVID-19 queries is compared to documents, and a series of postprocessing methods is applied to the initial ranking list to improve the match between the query and the biomedical information source and boost the position of relevant documents. Results: The methodology was evaluated in the context of the TREC-COVID challenge, achieving competitive results with the top-ranking teams participating in the competition. Particularly, the combination of bag-of-words and deep neural language models significantly outperformed an Okapi Best Match 25?based baseline, retrieving on average, 83% of relevant documents in the top 20. Conclusions: These results indicate that multistage retrieval supported by deep learning could enhance identification of literature for COVID-19?related questions posed using natural language. UR - https://www.jmir.org/2021/9/e30161 UR - http://dx.doi.org/10.2196/30161 UR - http://www.ncbi.nlm.nih.gov/pubmed/34375298 ID - info:doi/10.2196/30161 ER - TY - JOUR AU - Lee, Yun Dong AU - Park, Jimyung AU - Noh, Sung Jai AU - Roh, Woong Hyun AU - Ha, Ho Jae AU - Lee, Young Eun AU - Son, Joon Sang AU - Park, Woong Rae PY - 2021/9/3 TI - Characteristics of Dimensional Psychopathology in Suicidal Patients With Major Psychiatric Disorders and Its Association With the Length of Hospital Stay: Algorithm Validation Study JO - JMIR Ment Health SP - e30827 VL - 8 IS - 9 KW - suicide KW - computed phenotype KW - natural language processing KW - research domain criteria KW - electronic health record N2 - Background: Suicide has emerged as a serious concern for public health; however, only few studies have revealed the differences between major psychiatric disorders and suicide. Recent studies have attempted to quantify research domain criteria (RDoC) into numeric scores to systematically use them in computerized methods. The RDoC scores were used to reveal the characteristics of suicide and its association with major psychiatric disorders. Objective: We intended to investigate the differences in the dimensional psychopathology among hospitalized suicidal patients and the association between the dimensional psychopathology of psychiatric disorders and length of hospital stay. Methods: This retrospective study enrolled hospitalized suicidal patients diagnosed with major psychiatric disorders (depression, schizophrenia, and bipolar disorder) between January 2010 and December 2020 at a tertiary hospital in South Korea. The RDoC scores were calculated using the patients? admission notes. To measure the differences between psychiatric disorder cohorts, analysis of variance and the Cochran Q test were conducted and post hoc analysis for RDoC domains was performed with the independent two-sample t test. A linear regression model was used to analyze the association between the RDoC scores and sociodemographic features and comorbidity index. To estimate the association between the RDoC scores and length of hospital stay, multiple logistic regression models were applied to each psychiatric disorder group. Results: We retrieved 732 admissions for 571 patients (465 with depression, 73 with schizophrenia, and 33 with bipolar disorder). We found significant differences in the dimensional psychopathology according to the psychiatric disorders. The patient group with depression showed the highest negative RDoC domain scores. In the cognitive and social RDoC domains, the groups with schizophrenia and bipolar disorder scored higher than the group with depression. In the arousal RDoC domain, the depression and bipolar disorder groups scored higher than the group with schizophrenia. We identified significant associations between the RDoC scores and length of stay for the depression and bipolar disorder groups. The odds ratios (ORs) of the length of stay were increased because of the higher negative RDoC domain scores in the group with depression (OR 1.058, 95% CI 1.006-1.114) and decreased by higher arousal RDoC domain scores in the group with bipolar disorder (OR 0.537, 95% CI 0.285-0.815). Conclusions: This study showed the association between the dimensional psychopathology of major psychiatric disorders related to suicide and the length of hospital stay and identified differences in the dimensional psychopathology of major psychiatric disorders. This may provide new perspectives for understanding suicidal patients. UR - https://mental.jmir.org/2021/9/e30827 UR - http://dx.doi.org/10.2196/30827 UR - http://www.ncbi.nlm.nih.gov/pubmed/34477555 ID - info:doi/10.2196/30827 ER - TY - JOUR AU - Chen, Pei-Fu AU - Wang, Ssu-Ming AU - Liao, Wei-Chih AU - Kuo, Lu-Cheng AU - Chen, Kuan-Chih AU - Lin, Yu-Cheng AU - Yang, Chi-Yu AU - Chiu, Chi-Hao AU - Chang, Shu-Chih AU - Lai, Feipei PY - 2021/8/31 TI - Automatic ICD-10 Coding and Training System: Deep Neural Network Based on Supervised Learning JO - JMIR Med Inform SP - e23230 VL - 9 IS - 8 KW - natural language processing KW - deep learning KW - International Classification of Diseases KW - Recurrent Neural Network KW - text classification N2 - Background: The International Classification of Diseases (ICD) code is widely used as the reference in medical system and billing purposes. However, classifying diseases into ICD codes still mainly relies on humans reading a large amount of written material as the basis for coding. Coding is both laborious and time-consuming. Since the conversion of ICD-9 to ICD-10, the coding task became much more complicated, and deep learning? and natural language processing?related approaches have been studied to assist disease coders. Objective: This paper aims at constructing a deep learning model for ICD-10 coding, where the model is meant to automatically determine the corresponding diagnosis and procedure codes based solely on free-text medical notes to improve accuracy and reduce human effort. Methods: We used diagnosis records of the National Taiwan University Hospital as resources and apply natural language processing techniques, including global vectors, word to vectors, embeddings from language models, bidirectional encoder representations from transformers, and single head attention recurrent neural network, on the deep neural network architecture to implement ICD-10 auto-coding. Besides, we introduced the attention mechanism into the classification model to extract the keywords from diagnoses and visualize the coding reference for training freshmen in ICD-10. Sixty discharge notes were randomly selected to examine the change in the F1-score and the coding time by coders before and after using our model. Results: In experiments on the medical data set of National Taiwan University Hospital, our prediction results revealed F1-scores of 0.715 and 0.618 for the ICD-10 Clinical Modification code and Procedure Coding System code, respectively, with a bidirectional encoder representations from transformers embedding approach in the Gated Recurrent Unit classification model. The well-trained models were applied on the ICD-10 web service for coding and training to ICD-10 users. With this service, coders can code with the F1-score significantly increased from a median of 0.832 to 0.922 (P<.05), but not in a reduced interval. Conclusions: The proposed model significantly improved the F1-score but did not decrease the time consumed in coding by disease coders. UR - https://medinform.jmir.org/2021/8/e23230 UR - http://dx.doi.org/10.2196/23230 UR - http://www.ncbi.nlm.nih.gov/pubmed/34463639 ID - info:doi/10.2196/23230 ER - TY - JOUR AU - Jing, Xia PY - 2021/8/27 TI - The Unified Medical Language System at 30 Years and How It Is Used and Published: Systematic Review and Content Analysis JO - JMIR Med Inform SP - e20675 VL - 9 IS - 8 KW - Unified Medical Language System KW - systematic literature analysis KW - biomedical informatics KW - health informatics N2 - Background: The Unified Medical Language System (UMLS) has been a critical tool in biomedical and health informatics, and the year 2021 marks its 30th anniversary. The UMLS brings together many broadly used vocabularies and standards in the biomedical field to facilitate interoperability among different computer systems and applications. Objective: Despite its longevity, there is no comprehensive publication analysis of the use of the UMLS. Thus, this review and analysis is conducted to provide an overview of the UMLS and its use in English-language peer-reviewed publications, with the objective of providing a comprehensive understanding of how the UMLS has been used in English-language peer-reviewed publications over the last 30 years. Methods: PubMed, ACM Digital Library, and the Nursing & Allied Health Database were used to search for studies. The primary search strategy was as follows: UMLS was used as a Medical Subject Headings term or a keyword or appeared in the title or abstract. Only English-language publications were considered. The publications were screened first, then coded and categorized iteratively, following the grounded theory. The review process followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. Results: A total of 943 publications were included in the final analysis. Moreover, 32 publications were categorized into 2 categories; hence the total number of publications before duplicates are removed is 975. After analysis and categorization of the publications, UMLS was found to be used in the following emerging themes or areas (the number of publications and their respective percentages are given in parentheses): natural language processing (230/975, 23.6%), information retrieval (125/975, 12.8%), terminology study (90/975, 9.2%), ontology and modeling (80/975, 8.2%), medical subdomains (76/975, 7.8%), other language studies (53/975, 5.4%), artificial intelligence tools and applications (46/975, 4.7%), patient care (35/975, 3.6%), data mining and knowledge discovery (25/975, 2.6%), medical education (20/975, 2.1%), degree-related theses (13/975, 1.3%), digital library (5/975, 0.5%), and the UMLS itself (150/975, 15.4%), as well as the UMLS for other purposes (27/975, 2.8%). Conclusions: The UMLS has been used successfully in patient care, medical education, digital libraries, and software development, as originally planned, as well as in degree-related theses, the building of artificial intelligence tools, data mining and knowledge discovery, foundational work in methodology, and middle layers that may lead to advanced products. Natural language processing, the UMLS itself, and information retrieval are the 3 most common themes that emerged among the included publications. The results, although largely related to academia, demonstrate that UMLS achieves its intended uses successfully, in addition to achieving uses broadly beyond its original intentions. UR - https://medinform.jmir.org/2021/8/e20675 UR - http://dx.doi.org/10.2196/20675 UR - http://www.ncbi.nlm.nih.gov/pubmed/34236337 ID - info:doi/10.2196/20675 ER - TY - JOUR AU - Manabe, Masae AU - Liew, Kongmeng AU - Yada, Shuntaro AU - Wakamiya, Shoko AU - Aramaki, Eiji PY - 2021/8/12 TI - Estimation of Psychological Distress in Japanese Youth Through Narrative Writing: Text-Based Stylometric and Sentiment Analyses JO - JMIR Form Res SP - e29500 VL - 5 IS - 8 KW - psychological distress KW - youth KW - narratives KW - natural language processing KW - Japan KW - mental health KW - stress KW - distress KW - young adult KW - teenager KW - sentiment N2 - Background: Internalizing mental illnesses associated with psychological distress are often underdetected. Text-based detection using natural language processing (NLP) methods is increasingly being used to complement conventional detection efforts. However, these approaches often rely on self-disclosure through autobiographical narratives that may not always be possible, especially in the context of the collectivistic Japanese culture. Objective: We propose the use of narrative writing as an alternative resource for mental illness detection in youth. Accordingly, in this study, we investigated the textual characteristics of narratives written by youth with psychological distress; our research focuses on the detection of psychopathological tendencies in written imaginative narratives. Methods: Using NLP tools such as stylometric measures and lexicon-based sentiment analysis, we examined short narratives from 52 Japanese youth (mean age 19.8 years, SD 3.1) obtained through crowdsourcing. Participants wrote a short narrative introduction to an imagined story before completing a questionnaire to quantify their tendencies toward psychological distress. Based on this score, participants were categorized into higher distress and lower distress groups. The written narratives were then analyzed using NLP tools and examined for between-group differences. Although outside the scope of this study, we also carried out a supplementary analysis of narratives written by adults using the same procedure. Results: Youth demonstrating higher tendencies toward psychological distress used significantly more positive (happiness-related) words, revealing differences in valence of the narrative content. No other significant differences were observed between the high and low distress groups. Conclusions: Youth with tendencies toward mental illness were found to write more positive stories that contained more happiness-related terms. These results may potentially have widespread implications on psychological distress screening on online platforms, particularly in cultures such as Japan that are not accustomed to self-disclosure. Although the mechanisms that we propose in explaining our results are speculative, we believe that this interpretation paves the way for future research in online surveillance and detection efforts. UR - https://formative.jmir.org/2021/8/e29500 UR - http://dx.doi.org/10.2196/29500 UR - http://www.ncbi.nlm.nih.gov/pubmed/34387556 ID - info:doi/10.2196/29500 ER - TY - JOUR AU - Stojanov, Riste AU - Popovski, Gorjan AU - Cenikj, Gjorgjina AU - Korou?i? Seljak, Barbara AU - Eftimov, Tome PY - 2021/8/9 TI - A Fine-Tuned Bidirectional Encoder Representations From Transformers Model for Food Named-Entity Recognition: Algorithm Development and Validation JO - J Med Internet Res SP - e28229 VL - 23 IS - 8 KW - food information extraction KW - named-entity recognition KW - fine-tuning BERT KW - semantic annotation KW - information extraction KW - BERT KW - bidirectional encoder representations from transformers KW - natural language processing KW - machine learning N2 - Background: Recently, food science has been garnering a lot of attention. There are many open research questions on food interactions, as one of the main environmental factors, with other health-related entities such as diseases, treatments, and drugs. In the last 2 decades, a large amount of work has been done in natural language processing and machine learning to enable biomedical information extraction. However, machine learning in food science domains remains inadequately resourced, which brings to attention the problem of developing methods for food information extraction. There are only few food semantic resources and few rule-based methods for food information extraction, which often depend on some external resources. However, an annotated corpus with food entities along with their normalization was published in 2019 by using several food semantic resources. Objective: In this study, we investigated how the recently published bidirectional encoder representations from transformers (BERT) model, which provides state-of-the-art results in information extraction, can be fine-tuned for food information extraction. Methods: We introduce FoodNER, which is a collection of corpus-based food named-entity recognition methods. It consists of 15 different models obtained by fine-tuning 3 pretrained BERT models on 5 groups of semantic resources: food versus nonfood entity, 2 subsets of Hansard food semantic tags, FoodOn semantic tags, and Systematized Nomenclature of Medicine Clinical Terms food semantic tags. Results: All BERT models provided very promising results with 93.30% to 94.31% macro F1 scores in the task of distinguishing food versus nonfood entity, which represents the new state-of-the-art technology in food information extraction. Considering the tasks where semantic tags are predicted, all BERT models obtained very promising results once again, with their macro F1 scores ranging from 73.39% to 78.96%. Conclusions: FoodNER can be used to extract and annotate food entities in 5 different tasks: food versus nonfood entities and distinguishing food entities on the level of food groups by using the closest Hansard semantic tags, the parent Hansard semantic tags, the FoodOn semantic tags, or the Systematized Nomenclature of Medicine Clinical Terms semantic tags. UR - https://www.jmir.org/2021/8/e28229 UR - http://dx.doi.org/10.2196/28229 UR - http://www.ncbi.nlm.nih.gov/pubmed/34383671 ID - info:doi/10.2196/28229 ER - TY - JOUR AU - Aerts, Hannelore AU - Kalra, Dipak AU - Sáez, Carlos AU - Ramírez-Anguita, Manuel Juan AU - Mayer, Miguel-Angel AU - Garcia-Gomez, M. Juan AU - Durà-Hernández, Marta AU - Thienpont, Geert AU - Coorevits, Pascal PY - 2021/8/4 TI - Quality of Hospital Electronic Health Record (EHR) Data Based on the International Consortium for Health Outcomes Measurement (ICHOM) in Heart Failure: Pilot Data Quality Assessment Study JO - JMIR Med Inform SP - e27842 VL - 9 IS - 8 KW - data quality KW - electronic health records KW - heart failure KW - value-based health insurance KW - patient outcome assessment N2 - Background: There is increasing recognition that health care providers need to focus attention, and be judged against, the impact they have on the health outcomes experienced by patients. The measurement of health outcomes as a routine part of clinical documentation is probably the only scalable way of collecting outcomes evidence, since secondary data collection is expensive and error-prone. However, there is uncertainty about whether routinely collected clinical data within electronic health record (EHR) systems includes the data most relevant to measuring and comparing outcomes and if those items are collected to a good enough data quality to be relied upon for outcomes assessment, since several studies have pointed out significant issues regarding EHR data availability and quality. Objective: In this paper, we first describe a practical approach to data quality assessment of health outcomes, based on a literature review of existing frameworks for quality assessment of health data and multistakeholder consultation. Adopting this approach, we performed a pilot study on a subset of 21 International Consortium for Health Outcomes Measurement (ICHOM) outcomes data items from patients with congestive heart failure. Methods: All available registries compatible with the diagnosis of heart failure within an EHR data repository of a general hospital (142,345 visits and 12,503 patients) were extracted and mapped to the ICHOM format. We focused our pilot assessment on 5 commonly used data quality dimensions: completeness, correctness, consistency, uniqueness, and temporal stability. Results: We found high scores (>95%) for the consistency, completeness, and uniqueness dimensions. Temporal stability analyses showed some changes over time in the reported use of medication to treat heart failure, as well as in the recording of past medical conditions. Finally, the investigation of data correctness suggested several issues concerning the characterization of missing data values. Many of these issues appear to be introduced while mapping the IMASIS-2 relational database contents to the ICHOM format, as the latter requires a level of detail that is not explicitly available in the coded data of an EHR. Conclusions: Overall, results of this pilot study revealed good data quality for the subset of heart failure outcomes collected at the Hospital del Mar. Nevertheless, some important data errors were identified that were caused by fundamentally different data collection practices in routine clinical care versus research, for which the ICHOM standard set was originally developed. To truly examine to what extent hospitals today are able to routinely collect the evidence of their success in achieving good health outcomes, future research would benefit from performing more extensive data quality assessments, including all data items from the ICHOM standards set and across multiple hospitals. UR - https://medinform.jmir.org/2021/8/e27842 UR - http://dx.doi.org/10.2196/27842 UR - http://www.ncbi.nlm.nih.gov/pubmed/34346902 ID - info:doi/10.2196/27842 ER - TY - JOUR AU - He, Kai AU - Yao, Lixia AU - Zhang, JiaWei AU - Li, Yufei AU - Li, Chen PY - 2021/8/4 TI - Construction of Genealogical Knowledge Graphs From Obituaries: Multitask Neural Network Extraction System JO - J Med Internet Res SP - e25670 VL - 23 IS - 8 KW - genealogical knowledge graph KW - EHR KW - information extraction KW - genealogy KW - neural network N2 - Background: Genealogical information, such as that found in family trees, is imperative for biomedical research such as disease heritability and risk prediction. Researchers have used policyholder and their dependent information in medical claims data and emergency contacts in electronic health records (EHRs) to infer family relationships at a large scale. We have previously demonstrated that online obituaries can be a novel data source for building more complete and accurate family trees. Objective: Aiming at supplementing EHR data with family relationships for biomedical research, we built an end-to-end information extraction system using a multitask-based artificial neural network model to construct genealogical knowledge graphs (GKGs) from online obituaries. GKGs are enriched family trees with detailed information including age, gender, death and birth dates, and residence. Methods: Built on a predefined family relationship map consisting of 4 types of entities (eg, people?s name, residence, birth date, and death date) and 71 types of relationships, we curated a corpus containing 1700 online obituaries from the metropolitan area of Minneapolis and St Paul in Minnesota. We also adopted data augmentation technology to generate additional synthetic data to alleviate the issue of data scarcity for rare family relationships. A multitask-based artificial neural network model was then built to simultaneously detect names, extract relationships between them, and assign attributes (eg, birth dates and death dates, residence, age, and gender) to each individual. In the end, we assemble related GKGs into larger ones by identifying people appearing in multiple obituaries. Results: Our system achieved satisfying precision (94.79%), recall (91.45%), and F-1 measures (93.09%) on 10-fold cross-validation. We also constructed 12,407 GKGs, with the largest one made up of 4 generations and 30 people. Conclusions: In this work, we discussed the meaning of GKGs for biomedical research, presented a new version of a corpus with a predefined family relationship map and augmented training data, and proposed a multitask deep neural system to construct and assemble GKGs. The results show our system can extract and demonstrate the potential of enriching EHR data for more genetic research. We share the source codes and system with the entire scientific community on GitHub without the corpus for privacy protection. UR - https://www.jmir.org/2021/8/e25670 UR - http://dx.doi.org/10.2196/25670 UR - http://www.ncbi.nlm.nih.gov/pubmed/34346903 ID - info:doi/10.2196/25670 ER - TY - JOUR AU - Fang, An AU - Lou, Pei AU - Hu, Jiahui AU - Zhao, Wanqing AU - Feng, Ming AU - Ren, Huiling AU - Chen, Xianlai PY - 2021/7/22 TI - Head and Tail Entity Fusion Model in Medical Knowledge Graph Construction: Case Study for Pituitary Adenoma JO - JMIR Med Inform SP - e28218 VL - 9 IS - 7 KW - knowledge graph KW - pituitary adenoma KW - entity fusion KW - similarity calculation N2 - Background: Pituitary adenoma is one of the most common central nervous system tumors. The diagnosis and treatment of pituitary adenoma remain very difficult. Misdiagnosis and recurrence often occur, and experienced neurosurgeons are in serious shortage. A knowledge graph can help interns quickly understand the medical knowledge related to pituitary tumor. Objective: The aim of this study was to develop a data fusion method suitable for medical data using data of pituitary adenomas integrated from different sources. The overall goal was to construct a knowledge graph for pituitary adenoma (KGPA) to be used for knowledge discovery. Methods: A complete framework suitable for the construction of a medical knowledge graph was developed, which was used to build the KGPA. The schema of the KGPA was manually constructed. Information of pituitary adenoma was automatically extracted from Chinese electronic medical records (CEMRs) and medical websites through a conditional random field model and newly designed web wrappers. An entity fusion method is proposed based on the head-and-tail entity fusion model to fuse the data from heterogeneous sources. Results: Data were extracted from 300 CEMRs of pituitary adenoma and 4 health portals. Entity fusion was carried out using the proposed data fusion model. The F1 scores of the head and tail entity fusions were 97.32% and 98.57%, respectively. Triples from the constructed KGPA were selected for evaluation, demonstrating 95.4% accuracy. Conclusions: This paper introduces an approach to fuse triples extracted from heterogeneous data sources, which can be used to build a knowledge graph. The evaluation results showed that the data in the KGPA are of high quality. The constructed KGPA can help physicians in clinical practice. UR - https://medinform.jmir.org/2021/7/e28218 UR - http://dx.doi.org/10.2196/28218 UR - http://www.ncbi.nlm.nih.gov/pubmed/34057414 ID - info:doi/10.2196/28218 ER - TY - JOUR AU - Andy, Anietie PY - 2021/7/20 TI - Studying How Individuals Who Express the Feeling of Loneliness in an Online Loneliness Forum Communicate in a Nonloneliness Forum: Observational Study JO - JMIR Form Res SP - e28738 VL - 5 IS - 7 KW - loneliness KW - Reddit KW - nonloneliness KW - mental health KW - eHealth KW - forum KW - online forum KW - communication KW - natural language processing KW - language KW - linguistics N2 - Background: Loneliness is a public health concern, and increasingly, individuals experiencing loneliness are seeking support on online forums, some of which focus on discussions around loneliness (loneliness forums). Some of these individuals may also seek support around loneliness on online forums not related to loneliness or well-being (nonloneliness forums). Hence, to design and implement appropriate and efficient online loneliness interventions, it is important to understand how individuals who express and seek support around loneliness on online loneliness forums communicate in nonloneliness forums; this could provide further insights into the support needs and concerns of these users. Objective: This study aims to explore how users who express the feeling of loneliness and seek support around loneliness on an online loneliness forum communicate in an online nonloneliness forum. Methods: A total of 2401 users who expressed loneliness in posts published on a loneliness forum on Reddit and had published posts in a nonloneliness forum were identified. Using latent Dirichlet allocation (a natural language processing algorithm); Linguistic Inquiry and Word Count (a psycholinguistic dictionary); and the word score?based language features valence, arousal, and dominance, the language use differences in posts published in the nonloneliness forum by these users compared to a control group of users who did not belong to any loneliness forum on Reddit were determined. Results: It was found that in posts published in the nonloneliness forum, users who expressed loneliness tend to use more words associated with the Linguistic Inquiry and Word Count categories on sadness (Cohen d=0.10) and seeking to socialize (Cohen d=0.114), and use words associated with valence (Cohen d=0.364) and dominance (Cohen d=0.117). In addition, they tend to publish posts related to latent Dirichlet allocation topics such as relationships (Cohen d=0.105) and family and friends and mental health (Cohen d=0.10). Conclusions: There are clear distinctions in language use in nonloneliness forum posts by users who express loneliness compared to a control group of users. These findings can help with the design and implementation of online interventions around loneliness. UR - https://formative.jmir.org/2021/7/e28738 UR - http://dx.doi.org/10.2196/28738 UR - http://www.ncbi.nlm.nih.gov/pubmed/34283026 ID - info:doi/10.2196/28738 ER - TY - JOUR AU - Fecho, Karamarie AU - Bizon, Chris AU - Miller, Frederick AU - Schurman, Shepherd AU - Schmitt, Charles AU - Xue, William AU - Morton, Kenneth AU - Wang, Patrick AU - Tropsha, Alexander PY - 2021/7/20 TI - A Biomedical Knowledge Graph System to Propose Mechanistic Hypotheses for Real-World Environmental Health Observations: Cohort Study and Informatics Application JO - JMIR Med Inform SP - e26714 VL - 9 IS - 7 KW - knowledge graph KW - knowledge representation KW - data exploration KW - generalizability KW - discovery KW - open science KW - immune-mediated disease N2 - Background: Knowledge graphs are a common form of knowledge representation in biomedicine and many other fields. We developed an open biomedical knowledge graph?based system termed Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP). ROBOKOP consists of both a front-end user interface and a back-end knowledge graph. The ROBOKOP user interface allows users to posit questions and explore answer subgraphs. Users can also posit questions through direct Cypher query of the underlying knowledge graph, which currently contains roughly 6 million nodes or biomedical entities and 140 million edges or predicates describing the relationship between nodes, drawn from over 30 curated data sources. Objective: We aimed to apply ROBOKOP to survey data on workplace exposures and immune-mediated diseases from the Environmental Polymorphisms Registry (EPR) within the National Institute of Environmental Health Sciences. Methods: We analyzed EPR survey data and identified 45 associations between workplace chemical exposures and immune-mediated diseases, as self-reported by study participants (n= 4574), with 20 associations significant at P<.05 after false discovery rate correction. We then used ROBOKOP to (1) validate the associations by determining whether plausible connections exist within the ROBOKOP knowledge graph and (2) propose biological mechanisms that might explain them and serve as hypotheses for subsequent testing. We highlight the following three exemplar associations: carbon monoxide-multiple sclerosis, ammonia-asthma, and isopropanol-allergic disease. Results: ROBOKOP successfully returned answer sets for three queries that were posed in the context of the driving examples. The answer sets included potential intermediary genes, as well as supporting evidence that might explain the observed associations. Conclusions: We demonstrate real-world application of ROBOKOP to generate mechanistic hypotheses for associations between workplace chemical exposures and immune-mediated diseases. We expect that ROBOKOP will find broad application across many biomedical fields and other scientific disciplines due to its generalizability, speed to discovery and generation of mechanistic hypotheses, and open nature. UR - https://medinform.jmir.org/2021/7/e26714 UR - http://dx.doi.org/10.2196/26714 UR - http://www.ncbi.nlm.nih.gov/pubmed/34283031 ID - info:doi/10.2196/26714 ER - TY - JOUR AU - Yu, Shun AU - Le, Anh AU - Feld, Emily AU - Schriver, Emily AU - Gabriel, Peter AU - Doucette, Abigail AU - Narayan, Vivek AU - Feldman, Michael AU - Schwartz, Lauren AU - Maxwell, Kara AU - Mowery, Danielle PY - 2021/7/2 TI - A Natural Language Processing?Assisted Extraction System for Gleason Scores: Development and Usability Study JO - JMIR Cancer SP - e27970 VL - 7 IS - 3 KW - NLP KW - Gleason score KW - prostate cancer KW - natural language processing N2 - Background: Natural language processing (NLP) offers significantly faster variable extraction compared to traditional human extraction but cannot interpret complicated notes as well as humans can. Thus, we hypothesized that an ?NLP-assisted? extraction system, which uses humans for complicated notes and NLP for uncomplicated notes, could produce faster extraction without compromising accuracy. Objective: The aim of this study was to develop and pilot an NLP-assisted extraction system to leverage the strengths of both human and NLP extraction of prostate cancer Gleason scores. Methods: We collected all available clinical and pathology notes for prostate cancer patients in an unselected academic biobank cohort. We developed an NLP system to extract prostate cancer Gleason scores from both clinical and pathology notes. Next, we designed and implemented the NLP-assisted extraction system algorithm to categorize notes into ?uncomplicated? and ?complicated? notes. Uncomplicated notes were assigned to NLP extraction and complicated notes were assigned to human extraction. We randomly reviewed 200 patients to assess the accuracy and speed of our NLP-assisted extraction system and compared it to NLP extraction alone and human extraction alone. Results: Of the 2051 patients in our cohort, the NLP system extracted a prostate surgery Gleason score from 1147 (55.92%) patients and a prostate biopsy Gleason score from 1624 (79.18%) patients. Our NLP-assisted extraction system had an overall accuracy rate of 98.7%, which was similar to the accuracy of human extraction alone (97.5%; P=.17) and significantly higher than the accuracy of NLP extraction alone (95.3%; P<.001). Moreover, our NLP-assisted extraction system reduced the workload of human extractors by approximately 95%, resulting in an average extraction time of 12.7 seconds per patient (vs 256.1 seconds per patient for human extraction alone). Conclusions: We demonstrated that an NLP-assisted extraction system was able to achieve much faster Gleason score extraction compared to traditional human extraction without sacrificing accuracy. UR - https://cancer.jmir.org/2021/3/e27970 UR - http://dx.doi.org/10.2196/27970 UR - http://www.ncbi.nlm.nih.gov/pubmed/34255641 ID - info:doi/10.2196/27970 ER - TY - JOUR AU - Yum, Yunjin AU - Lee, Moon Jeong AU - Jang, Joung Moon AU - Kim, Yoojoong AU - Kim, Jong-Ho AU - Kim, Seongtae AU - Shin, Unsub AU - Song, Sanghoun AU - Joo, Joon Hyung PY - 2021/6/24 TI - A Word Pair Dataset for Semantic Similarity and Relatedness in Korean Medical Vocabulary: Reference Development and Validation JO - JMIR Med Inform SP - e29667 VL - 9 IS - 6 KW - medical word pair KW - similarity KW - relatedness KW - word embedding KW - fastText KW - Korean N2 - Background: The fact that medical terms require special expertise and are becoming increasingly complex makes it difficult to employ natural language processing techniques in medical informatics. Several human-validated reference standards for medical terms have been developed to evaluate word embedding models using the semantic similarity and relatedness of medical word pairs. However, there are very few reference standards in non-English languages. In addition, because the existing reference standards were developed a long time ago, there is a need to develop an updated standard to represent recent findings in medical sciences. Objective: We propose a new Korean word pair reference set to verify embedding models. Methods: From January 2010 to December 2020, 518 medical textbooks, 72,844 health information news, and 15,698 medical research articles were collected, and the top 10,000 medical terms were selected to develop medical word pairs. Attending physicians (n=16) participated in the verification of the developed set with 607 word pairs. Results: The proportion of word pairs answered by all participants was 90.8% (551/607) for the similarity task and 86.5% (525/605) for the relatedness task. The similarity and relatedness of the word pair showed a high correlation (?=0.70, P<.001). The intraclass correlation coefficients to assess the interrater agreements of the word pair sets were 0.47 on the similarity task and 0.53 on the relatedness task. The final reference standard was 604 word pairs for the similarity task and 599 word pairs for relatedness, excluding word pairs with answers corresponding to outliers and word pairs that were answered by less than 50% of all the respondents. When FastText models were applied to the final reference standard word pair sets, the embedding models learning medical documents had a higher correlation between the calculated cosine similarity scores compared to human-judged similarity and relatedness scores (namu, ?=0.12 vs with medical text for the similarity task, ?=0.47; namu, ?=0.02 vs with medical text for the relatedness task, ?=0.30). Conclusions: Korean medical word pair reference standard sets for semantic similarity and relatedness were developed based on medical documents from the past 10 years. It is expected that our word pair reference sets will be actively utilized in the development of medical and multilingual natural language processing technology in the future. UR - https://medinform.jmir.org/2021/6/e29667/ UR - http://dx.doi.org/10.2196/29667 UR - http://www.ncbi.nlm.nih.gov/pubmed/34185005 ID - info:doi/10.2196/29667 ER - TY - JOUR AU - Wang, Meng AU - Wang, Haofen AU - Liu, Xing AU - Ma, Xinyu AU - Wang, Beilun PY - 2021/6/19 TI - Drug-Drug Interaction Predictions via Knowledge Graph and Text Embedding: Instrument Validation Study JO - JMIR Med Inform SP - e28277 VL - 9 IS - 6 KW - drug-drug interactions KW - knowledge graph KW - natural language processing N2 - Background: Minimizing adverse reactions caused by drug-drug interactions (DDIs) has always been a prominent research topic in clinical pharmacology. Detecting all possible interactions through clinical studies before a drug is released to the market is a demanding task. The power of big data is opening up new approaches to discovering various DDIs. However, these data contain a huge amount of noise and provide knowledge bases that are far from being complete or used with reliability. Most existing studies focus on predicting binary DDIs between drug pairs and ignore other interactions. Objective: Leveraging both drug knowledge graphs and biomedical text is a promising pathway for rich and comprehensive DDI prediction, but it is not without issues. Our proposed model seeks to address the following challenges: data noise and incompleteness, data sparsity, and computational complexity. Methods: We propose a novel framework, Predicting Rich DDI, to predict DDIs. The framework uses graph embedding to overcome data incompleteness and sparsity issues to make multiple DDI label predictions. First, a large-scale drug knowledge graph is generated from different sources. The knowledge graph is then embedded with comprehensive biomedical text into a common low-dimensional space. Finally, the learned embeddings are used to efficiently compute rich DDI information through a link prediction process. Results: To validate the effectiveness of the proposed framework, extensive experiments were conducted on real-world data sets. The results demonstrate that our model outperforms several state-of-the-art baseline methods in terms of capability and accuracy. Conclusions: We propose a novel framework, Predicting Rich DDI, to predict DDIs. Using rich DDI information, it can competently predict multiple labels for a pair of drugs across numerous domains, ranging from pharmacological mechanisms to side effects. To the best of our knowledge, this framework is the first to provide a joint translation-based embedding model that learns DDIs by integrating drug knowledge graphs and biomedical text simultaneously in a common low-dimensional space. The model also predicts DDIs using multiple labels rather than single or binary labels. Extensive experiments were conducted on real-world data sets to demonstrate the effectiveness and efficiency of the model. The results show our proposed framework outperforms several state-of-the-art baselines. UR - https://medinform.jmir.org/2021/6/e28277/ UR - http://dx.doi.org/10.2196/28277 UR - http://www.ncbi.nlm.nih.gov/pubmed/34185011 ID - info:doi/10.2196/28277 ER - TY - JOUR AU - Gaudet-Blavignac, Christophe AU - Raisaro, Louis Jean AU - Touré, Vasundra AU - Österle, Sabine AU - Crameri, Katrin AU - Lovis, Christian PY - 2021/6/24 TI - A National, Semantic-Driven, Three-Pillar Strategy to Enable Health Data Secondary Usage Interoperability for Research Within the Swiss Personalized Health Network: Methodological Study JO - JMIR Med Inform SP - e27591 VL - 9 IS - 6 KW - interoperability KW - clinical data reuse KW - personalized medicine N2 - Background: Interoperability is a well-known challenge in medical informatics. Current trends in interoperability have moved from a data model technocentric approach to sustainable semantics, formal descriptive languages, and processes. Despite many initiatives and investments for decades, the interoperability challenge remains crucial. The need for data sharing for most purposes ranging from patient care to secondary uses, such as public health, research, and quality assessment, faces unmet problems. Objective: This work was performed in the context of a large Swiss Federal initiative aiming at building a national infrastructure for reusing consented data acquired in the health care and research system to enable research in the field of personalized medicine in Switzerland. The initiative is the Swiss Personalized Health Network (SPHN). This initiative is providing funding to foster use and exchange of health-related data for research. As part of the initiative, a national strategy to enable a semantically interoperable clinical data landscape was developed and implemented. Methods: A deep analysis of various approaches to address interoperability was performed at the start, including large frameworks in health care, such as Health Level Seven (HL7) and Integrating Healthcare Enterprise (IHE), and in several domains, such as regulatory agencies (eg, Clinical Data Interchange Standards Consortium [CDISC]) and research communities (eg, Observational Medical Outcome Partnership [OMOP]), to identify bottlenecks and assess sustainability. Based on this research, a strategy composed of three pillars was designed. It has strong multidimensional semantics, descriptive formal language for exchanges, and as many data models as needed to comply with the needs of various communities. Results: This strategy has been implemented stepwise in Switzerland since the middle of 2019 and has been adopted by all university hospitals and high research organizations. The initiative is coordinated by a central organization, the SPHN Data Coordination Center of the SIB Swiss Institute of Bioinformatics. The semantics is mapped by domain experts on various existing standards, such as Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), Logical Observation Identifiers Names and Codes (LOINC), and International Classification of Diseases (ICD). The resource description framework (RDF) is used for storing and transporting data, and to integrate information from different sources and standards. Data transformers based on SPARQL query language are implemented to convert RDF representations to the numerous data models required by the research community or bridge with other systems, such as electronic case report forms. Conclusions: The SPHN strategy successfully implemented existing standards in a pragmatic and applicable way. It did not try to build any new standards but used existing ones in a nondogmatic way. It has now been funded for another 4 years, bringing the Swiss landscape into a new dimension to support research in the field of personalized medicine and large interoperable clinical data. UR - https://medinform.jmir.org/2021/6/e27591/ UR - http://dx.doi.org/10.2196/27591 UR - http://www.ncbi.nlm.nih.gov/pubmed/34185008 ID - info:doi/10.2196/27591 ER - TY - JOUR AU - Qin, Xuan AU - Yao, Xinzhi AU - Xia, Jingbo PY - 2021/6/18 TI - A Novel Metric to Quantify the Effect of Pathway Enrichment Evaluation With Respect to Biomedical Text-Mined Terms: Development and Feasibility Study JO - JMIR Med Inform SP - e28247 VL - 9 IS - 6 KW - pathway enrichment KW - metric KW - evaluation KW - text mining N2 - Background: Natural language processing has long been applied in various applications for biomedical knowledge inference and discovery. Enrichment analysis based on named entity recognition is a classic application for inferring enriched associations in terms of specific biomedical entities such as gene, chemical, and mutation. Objective: The aim of this study was to investigate the effect of pathway enrichment evaluation with respect to biomedical text-mining results and to develop a novel metric to quantify the effect. Methods: Four biomedical text mining methods were selected to represent natural language processing methods on drug-related gene mining. Subsequently, a pathway enrichment experiment was performed by using the mined genes, and a series of inverse pathway frequency (IPF) metrics was proposed accordingly to evaluate the effect of pathway enrichment. Thereafter, 7 IPF metrics and traditional P value metrics were compared in simulation experiments to test the robustness of the proposed metrics. Results: IPF metrics were evaluated in a case study of rapamycin-related gene set. By applying the best IPF metrics in a pathway enrichment simulation test, a novel discovery of drug efficacy of rapamycin for breast cancer was replicated from the data chosen prior to the year 2000. Our findings show the effectiveness of the best IPF metric in support of knowledge discovery in new drug use. Further, the mechanism underlying the drug-disease association was visualized by Cytoscape. Conclusions: The results of this study suggest the effectiveness of the proposed IPF metrics in pathway enrichment evaluation as well as its application in drug use discovery. UR - https://medinform.jmir.org/2021/6/e28247 UR - http://dx.doi.org/10.2196/28247 UR - http://www.ncbi.nlm.nih.gov/pubmed/34142969 ID - info:doi/10.2196/28247 ER - TY - JOUR AU - Deng, Lizong AU - Chen, Luming AU - Yang, Tao AU - Liu, Mi AU - Li, Shicheng AU - Jiang, Taijiao PY - 2021/6/15 TI - Constructing High-Fidelity Phenotype Knowledge Graphs for Infectious Diseases With a Fine-Grained Semantic Information Model: Development and Usability Study JO - J Med Internet Res SP - e26892 VL - 23 IS - 6 KW - knowledge graph KW - knowledge granularity KW - machine learning KW - high-fidelity phenotyping KW - phenotyping KW - phenotype KW - semantic N2 - Background: Phenotypes characterize the clinical manifestations of diseases and provide important information for diagnosis. Therefore, the construction of phenotype knowledge graphs for diseases is valuable to the development of artificial intelligence in medicine. However, phenotype knowledge graphs in current knowledge bases such as WikiData and DBpedia are coarse-grained knowledge graphs because they only consider the core concepts of phenotypes while neglecting the details (attributes) associated with these phenotypes. Objective: To characterize the details of disease phenotypes for clinical guidelines, we proposed a fine-grained semantic information model named PhenoSSU (semantic structured unit of phenotypes). Methods: PhenoSSU is an ?entity-attribute-value? model by its very nature, and it aims to capture the full semantic information underlying phenotype descriptions with a series of attributes and values. A total of 193 clinical guidelines for infectious diseases from Wikipedia were selected as the study corpus, and 12 attributes from SNOMED-CT were introduced into the PhenoSSU model based on the co-occurrences of phenotype concepts and attribute values. The expressive power of the PhenoSSU model was evaluated by analyzing whether PhenoSSU instances could capture the full semantics underlying the descriptions of the corresponding phenotypes. To automatically construct fine-grained phenotype knowledge graphs, a hybrid strategy that first recognized phenotype concepts with the MetaMap tool and then predicted the attribute values of phenotypes with machine learning classifiers was developed. Results: Fine-grained phenotype knowledge graphs of 193 infectious diseases were manually constructed with the BRAT annotation tool. A total of 4020 PhenoSSU instances were annotated in these knowledge graphs, and 3757 of them (89.5%) were found to be able to capture the full semantics underlying the descriptions of the corresponding phenotypes listed in clinical guidelines. By comparison, other information models, such as the clinical element model and the HL7 fast health care interoperability resource model, could only capture the full semantics underlying 48.4% (2034/4020) and 21.8% (914/4020) of the descriptions of phenotypes listed in clinical guidelines, respectively. The hybrid strategy achieved an F1-score of 0.732 for the subtask of phenotype concept recognition and an average weighted accuracy of 0.776 for the subtask of attribute value prediction. Conclusions: PhenoSSU is an effective information model for the precise representation of phenotype knowledge for clinical guidelines, and machine learning can be used to improve the efficiency of constructing PhenoSSU-based knowledge graphs. Our work will potentially shift the focus of medical knowledge engineering from a coarse-grained level to a more fine-grained level. UR - https://www.jmir.org/2021/6/e26892 UR - http://dx.doi.org/10.2196/26892 UR - http://www.ncbi.nlm.nih.gov/pubmed/34128811 ID - info:doi/10.2196/26892 ER - TY - JOUR AU - Lee, Ji-Hyun AU - Park, Hyeoun-Ae AU - Song, Tae-Min PY - 2021/6/14 TI - A Determinants-of-Fertility Ontology for Detecting Future Signals of Fertility Issues From Social Media Data: Development of an Ontology JO - J Med Internet Res SP - e25028 VL - 23 IS - 6 KW - ontology KW - fertility KW - public policy KW - South Korea KW - social media KW - future KW - infodemiology KW - infoveillance N2 - Background: South Korea has the lowest fertility rate in the world despite considerable governmental efforts to boost it. Increasing the fertility rate and achieving the desired outcomes of any implemented policies requires reliable data on the ongoing trends in fertility and preparations for the future based on these trends. Objective: The aims of this study were to (1) develop a determinants-of-fertility ontology with terminology for collecting and analyzing social media data; (2) determine the description logics, content coverage, and structural and representational layers of the ontology; and (3) use the ontology to detect future signals of fertility issues. Methods: An ontology was developed using the Ontology Development 101 methodology. The domain and scope of the ontology were defined by compiling a list of competency questions. The terms were collected from Korean government reports, Korea?s Basic Plan for Low Fertility and Aging Society, a national survey about marriage and childbirth, and social media postings on fertility issues. The classes and their hierarchy were defined using a top-down approach based on an ecological model. The internal structure of classes was defined using the entity-attribute-value model. The description logics of the ontology were evaluated using Protégé (version 5.5.0), and the content coverage was evaluated by comparing concepts extracted from social media posts with the list of ontology classes. The structural and representational layers of the ontology were evaluated by experts. Social media data were collected from 183 online channels between January 1, 2011, and June 30, 2015. To detect future signals of fertility issues, 2 classes of the ontology, the socioeconomic and cultural environment, and public policy, were identified as keywords. A keyword issue map was constructed, and the defined keywords were mapped to identify future signals. R software (version 3.5.2) was used to mine for future signals. Results: A determinants-of-fertility ontology comprised 236 classes and terminology comprised 1464 synonyms of the 236 classes. Concept classes in the ontology were found to be coherently and consistently defined. The ontology included more than 90% of the concepts that appeared in social media posts on fertility policies. Average scores for all of the criteria for structural and representations layers exceeded 4 on a 5-point scale. Violence and abuse (socioeconomic and cultural factor) and flexible working arrangement (fertility policy) were weak signals, suggesting that they could increase rapidly in the future. Conclusions: The determinants-of-fertility ontology developed in this study can be used as a framework for collecting and analyzing social media data on fertility issues and detecting future signals of fertility issues. The future signals identified in this study will be useful for policy makers who are developing policy responses to low fertility. UR - https://www.jmir.org/2021/6/e25028 UR - http://dx.doi.org/10.2196/25028 UR - http://www.ncbi.nlm.nih.gov/pubmed/34125068 ID - info:doi/10.2196/25028 ER - TY - JOUR AU - Jia, Qi AU - Zhang, Dezheng AU - Xu, Haifeng AU - Xie, Yonghong PY - 2021/6/14 TI - Extraction of Traditional Chinese Medicine Entity: Design of a Novel Span-Level Named Entity Recognition Method With Distant Supervision JO - JMIR Med Inform SP - e28219 VL - 9 IS - 6 KW - traditional Chinese medicine KW - named entity recognition KW - span level KW - distantly supervised N2 - Background: Traditional Chinese medicine (TCM) clinical records contain the symptoms of patients, diagnoses, and subsequent treatment of doctors. These records are important resources for research and analysis of TCM diagnosis knowledge. However, most of TCM clinical records are unstructured text. Therefore, a method to automatically extract medical entities from TCM clinical records is indispensable. Objective: Training a medical entity extracting model needs a large number of annotated corpus. The cost of annotated corpus is very high and there is a lack of gold-standard data sets for supervised learning methods. Therefore, we utilized distantly supervised named entity recognition (NER) to respond to the challenge. Methods: We propose a span-level distantly supervised NER approach to extract TCM medical entity. It utilizes the pretrained language model and a simple multilayer neural network as classifier to detect and classify entity. We also designed a negative sampling strategy for the span-level model. The strategy randomly selects negative samples in every epoch and filters the possible false-negative samples periodically. It reduces the bad influence from the false-negative samples. Results: We compare our methods with other baseline methods to illustrate the effectiveness of our method on a gold-standard data set. The F1 score of our method is 77.34 and it remarkably outperforms the other baselines. Conclusions: We developed a distantly supervised NER approach to extract medical entity from TCM clinical records. We estimated our approach on a TCM clinical record data set. Our experimental results indicate that the proposed approach achieves a better performance than other baselines. UR - https://medinform.jmir.org/2021/6/e28219 UR - http://dx.doi.org/10.2196/28219 UR - http://www.ncbi.nlm.nih.gov/pubmed/34125076 ID - info:doi/10.2196/28219 ER - TY - JOUR AU - Cha, Dongchul AU - Sung, MinDong AU - Park, Yu-Rang PY - 2021/6/9 TI - Implementing Vertical Federated Learning Using Autoencoders: Practical Application, Generalizability, and Utility Study JO - JMIR Med Inform SP - e26598 VL - 9 IS - 6 KW - federated learning KW - vertically incomplete data KW - privacy KW - machine learning KW - coding KW - data KW - performance KW - model KW - security KW - training KW - dataset KW - unsupervised learning KW - data sharing KW - protection N2 - Background: Machine learning (ML) is now widely deployed in our everyday lives. Building robust ML models requires a massive amount of data for training. Traditional ML algorithms require training data centralization, which raises privacy and data governance issues. Federated learning (FL) is an approach to overcome this issue. We focused on applying FL on vertically partitioned data, in which an individual?s record is scattered among different sites. Objective: The aim of this study was to perform FL on vertically partitioned data to achieve performance comparable to that of centralized models without exposing the raw data. Methods: We used three different datasets (Adult income, Schwannoma, and eICU datasets) and vertically divided each dataset into different pieces. Following the vertical division of data, overcomplete autoencoder-based model training was performed for each site. Following training, each site?s data were transformed into latent data, which were aggregated for training. A tabular neural network model with categorical embedding was used for training. A centrally based model was used as a baseline model, which was compared to that of FL in terms of accuracy and area under the receiver operating characteristic curve (AUROC). Results: The autoencoder-based network successfully transformed the original data into latent representations with no domain knowledge applied. These altered data were different from the original data in terms of the feature space and data distributions, indicating appropriate data security. The loss of performance was minimal when using an overcomplete autoencoder; accuracy loss was 1.2%, 8.89%, and 1.23%, and AUROC loss was 1.1%, 0%, and 1.12% in the Adult income, Schwannoma, and eICU dataset, respectively. Conclusions: We proposed an autoencoder-based ML model for vertically incomplete data. Since our model is based on unsupervised learning, no domain-specific knowledge is required in individual sites. Under the circumstances where direct data sharing is not available, our approach may be a practical solution enabling both data protection and building a robust model. UR - https://medinform.jmir.org/2021/6/e26598 UR - http://dx.doi.org/10.2196/26598 UR - http://www.ncbi.nlm.nih.gov/pubmed/34106083 ID - info:doi/10.2196/26598 ER - TY - JOUR AU - Kim, Gyungha AU - Jeon, Hwawoo AU - Park, Kee Sung AU - Choi, Suk Yong AU - Lim, Yoonseob PY - 2021/6/8 TI - A Care Knowledge Management System Based on an Ontological Model of Caring for People With Dementia: Knowledge Representation and Development Study JO - J Med Internet Res SP - e25968 VL - 23 IS - 6 KW - caregiver KW - caregiver for person with dementia KW - knowledge model KW - ontology KW - knowledge management KW - semantic reasoning N2 - Background: Caregivers of people with dementia find it extremely difficult to choose the best care method because of complex environments and the variable symptoms of dementia. To alleviate this care burden, interventions have been proposed that use computer- or web-based applications. For example, an automatic diagnosis of the condition can improve the well-being of both the person with dementia and the caregiver. Other interventions support the individual with dementia in living independently. Objective: The aim of this study was to develop an ontology-based care knowledge management system for people with dementia that will provide caregivers with a care guide suited to the environment and to the individual patient?s symptoms. This should also enable knowledge sharing among caregivers. Methods: To build the care knowledge model, we reviewed existing ontologies that contain concepts and knowledge descriptions relating to the care of those with dementia, and we considered dementia care manuals. The basic concepts of the care ontology were confirmed by experts in Korea. To infer the different care methods required for the individual dementia patient, the reasoning rules as defined in Semantic Web Rule Languages and Prolog were utilized. The accuracy of the care knowledge in the ontological model and the usability of the proposed system were evaluated by using the Pellet reasoner and OntOlogy Pitfall Scanner!, and a survey and interviews were conducted with caregivers working in care centers in Korea. Results: The care knowledge model contains six top-level concepts: care knowledge, task, assessment, person, environment, and medical knowledge. Based on this ontological model of dementia care, caregivers at a dementia care facility in Korea were able to access the care knowledge easily through a graphical user interface. The evaluation by the care experts showed that the system contained accurate care knowledge and a level of assessment comparable to normal assessment tools. Conclusions: In this study, we developed a care knowledge system that can provide caregivers with care guides suited to individuals with dementia. We anticipate that the system could reduce the workload of caregivers. UR - https://www.jmir.org/2021/6/e25968 UR - http://dx.doi.org/10.2196/25968 UR - http://www.ncbi.nlm.nih.gov/pubmed/34100762 ID - info:doi/10.2196/25968 ER - TY - JOUR AU - Park, Hyung AU - Song, Min AU - Lee, Byul Eun AU - Seo, Kyung Bo AU - Choi, Min Chang PY - 2021/5/17 TI - An Attention Model With Transfer Embeddings to Classify Pneumonia-Related Bilingual Imaging Reports: Algorithm Development and Validation JO - JMIR Med Inform SP - e24803 VL - 9 IS - 5 KW - deep learning KW - natural language process KW - attention KW - clinical data KW - pneumonia KW - classification KW - medical imaging KW - electronic health record KW - machine learning KW - model N2 - Background: In the analysis of electronic health records, proper labeling of outcomes is mandatory. To obtain proper information from radiologic reports, several studies were conducted to classify radiologic reports using deep learning. However, the classification of pneumonia in bilingual radiologic reports has not been conducted previously. Objective: The aim of this research was to classify radiologic reports into pneumonia or no pneumonia using a deep learning method. Methods: A data set of radiology reports for chest computed tomography and chest x-rays of surgical patients from January 2008 to January 2018 in the Asan Medical Center in Korea was retrospectively analyzed. The classification performance of our long short-term memory (LSTM)?Attention model was compared with various deep learning and machine learning methods. The area under the receiver operating characteristic curve (AUROC), area under the precision-recall curve, sensitivity, specificity, accuracy, and F1 score for the models were compared. Results: A total of 5450 radiologic reports were included that contained at least one pneumonia-related word. In the test set (n=1090), our proposed model showed 91.01% (992/1090) accuracy (AUROCs for negative, positive, and obscure were 0.98, 0.97, and 0.90, respectively). The top 3 performances of the models were based on FastText or LSTM. The convolutional neural network?based model showed a lower accuracy 73.03% (796/1090) than the other 2 algorithms. The classification of negative results had an F1 score of 0.96, whereas the classification of positive and uncertain results showed a lower performance (positive F1 score 0.83; uncertain F1 score 0.62). In the extra-validation set, our model showed 80.0% (642/803) accuracy (AUROCs for negative, positive, and obscure were 0.92, 0.96, and 0.84, respectively). Conclusions: Our method showed excellent performance in classifying pneumonia in bilingual radiologic reports. The method could enrich the research on pneumonia by obtaining exact outcomes from electronic health data. UR - https://medinform.jmir.org/2021/5/e24803 UR - http://dx.doi.org/10.2196/24803 UR - http://www.ncbi.nlm.nih.gov/pubmed/33820755 ID - info:doi/10.2196/24803 ER - TY - JOUR AU - Wang, Zheyu AU - An, Jiye AU - Lin, Hui AU - Zhou, Jiaqiang AU - Liu, Fang AU - Chen, Juan AU - Duan, Huilong AU - Deng, Ning PY - 2021/5/17 TI - Pathway-Driven Coordinated Telehealth System for Management of Patients With Single or Multiple Chronic Diseases in China: System Development and Retrospective Study JO - JMIR Med Inform SP - e27228 VL - 9 IS - 5 KW - chronic disease KW - telehealth system KW - integrated care KW - pathway KW - ontology N2 - Background: Integrated care enhanced with information technology has emerged as a means to transform health services to meet the long-term care needs of patients with chronic diseases. However, the feasibility of applying integrated care to the emerging ?three-manager? mode in China remains to be explored. Moreover, few studies have attempted to integrate multiple types of chronic diseases into a single system. Objective: The aim of this study was to develop a coordinated telehealth system that addresses the existing challenges of the ?three-manager? mode in China while supporting the management of single or multiple chronic diseases. Methods: The system was designed based on a tailored integrated care model. The model was constructed at the individual scale, mainly focusing on specifying the involved roles and responsibilities through a universal care pathway. A custom ontology was developed to represent the knowledge contained in the model. The system consists of a service engine for data storage and decision support, as well as different forms of clients for care providers and patients. Currently, the system supports management of three single chronic diseases (hypertension, type 2 diabetes mellitus, and chronic obstructive pulmonary disease) and one type of multiple chronic conditions (hypertension with type 2 diabetes mellitus). A retrospective study was performed based on the long-term observational data extracted from the database to evaluate system usability, treatment effect, and quality of care. Results: The retrospective analysis involved 6964 patients with chronic diseases and 249 care providers who have registered in our system since its deployment in 2015. A total of 519,598 self-monitoring records have been submitted by the patients. The engine could generate different types of records regularly based on the specific care pathway. Results of the comparison tests and causal inference showed that a part of patient outcomes improved after receiving management through the system, especially the systolic blood pressure of patients with hypertension (P<.001 in all comparison tests and an approximately 5 mmHg decrease after intervention via causal inference). A regional case study showed that the work efficiency of care providers differed among individuals. Conclusions: Our system has potential to provide effective management support for single or multiple chronic conditions simultaneously. The tailored closed-loop care pathway was feasible and effective under the ?three-manager? mode in China. One direction for future work is to introduce advanced artificial intelligence techniques to construct a more personalized care pathway. UR - https://medinform.jmir.org/2021/5/e27228 UR - http://dx.doi.org/10.2196/27228 UR - http://www.ncbi.nlm.nih.gov/pubmed/33998999 ID - info:doi/10.2196/27228 ER - TY - JOUR AU - Jin, Haomiao AU - Chien, Sandy AU - Meijer, Erik AU - Khobragade, Pranali AU - Lee, Jinkook PY - 2021/5/10 TI - Learning From Clinical Consensus Diagnosis in India to Facilitate Automatic Classification of Dementia: Machine Learning Study JO - JMIR Ment Health SP - e27113 VL - 8 IS - 5 KW - dementia KW - Alzheimer disease KW - machine learning KW - artificial intelligence KW - diagnosis KW - classification KW - India KW - model N2 - Background: The Harmonized Diagnostic Assessment of Dementia for the Longitudinal Aging Study in India (LASI-DAD) is the first and only nationally representative study on late-life cognition and dementia in India (n=4096). LASI-DAD obtained clinical consensus diagnosis of dementia for a subsample of 2528 respondents. Objective: This study develops a machine learning model that uses data from the clinical consensus diagnosis in LASI-DAD to support the classification of dementia status. Methods: Clinicians were presented with the extensive data collected from LASI-DAD, including sociodemographic information and health history of respondents, results from the screening tests of cognitive status, and information obtained from informant interviews. Based on the Clinical Dementia Rating (CDR) and using an online platform, clinicians individually evaluated each case and then reached a consensus diagnosis. A 2-step procedure was implemented to train several candidate machine learning models, which were evaluated using a separate test set for predictive accuracy measurement, including the area under receiver operating curve (AUROC), accuracy, sensitivity, specificity, precision, F1 score, and kappa statistic. The ultimate model was selected based on overall agreement as measured by kappa. We further examined the overall accuracy and agreement with the final consensus diagnoses between the selected machine learning model and individual clinicians who participated in the clinical consensus diagnostic process. Finally, we applied the selected model to a subgroup of LASI-DAD participants for whom the clinical consensus diagnosis was not obtained to predict their dementia status. Results: Among the 2528 individuals who received clinical consensus diagnosis, 192 (6.7% after adjusting for sampling weight) were diagnosed with dementia. All candidate machine learning models achieved outstanding discriminative ability, as indicated by AUROC >.90, and had similar accuracy and specificity (both around 0.95). The support vector machine model outperformed other models with the highest sensitivity (0.81), F1 score (0.72), and kappa (.70, indicating substantial agreement) and the second highest precision (0.65). As a result, the support vector machine was selected as the ultimate model. Further examination revealed that overall accuracy and agreement were similar between the selected model and individual clinicians. Application of the prediction model on 1568 individuals without clinical consensus diagnosis classified 127 individuals as living with dementia. After applying sampling weight, we can estimate the prevalence of dementia in the population as 7.4%. Conclusions: The selected machine learning model has outstanding discriminative ability and substantial agreement with a clinical consensus diagnosis of dementia. The model can serve as a computer model of the clinical knowledge and experience encoded in the clinical consensus diagnostic process and has many potential applications, including predicting missed dementia diagnoses and serving as a clinical decision support tool or virtual rater to assist diagnosis of dementia. UR - https://mental.jmir.org/2021/5/e27113 UR - http://dx.doi.org/10.2196/27113 UR - http://www.ncbi.nlm.nih.gov/pubmed/33970122 ID - info:doi/10.2196/27113 ER - TY - JOUR AU - Alfattni, Ghada AU - Belousov, Maksim AU - Peek, Niels AU - Nenadic, Goran PY - 2021/5/5 TI - Extracting Drug Names and Associated Attributes From Discharge Summaries: Text Mining Study JO - JMIR Med Inform SP - e24678 VL - 9 IS - 5 KW - information extraction KW - electronic health records KW - discharge summaries KW - natural language processing KW - medication prescriptions N2 - Background: Drug prescriptions are often recorded in free-text clinical narratives; making this information available in a structured form is important to support many health-related tasks. Although several natural language processing (NLP) methods have been proposed to extract such information, many challenges remain. Objective: This study evaluates the feasibility of using NLP and deep learning approaches for extracting and linking drug names and associated attributes identified in clinical free-text notes and presents an extensive error analysis of different methods. This study initiated with the participation in the 2018 National NLP Clinical Challenges (n2c2) shared task on adverse drug events and medication extraction. Methods: The proposed system (DrugEx) consists of a named entity recognizer (NER) to identify drugs and associated attributes and a relation extraction (RE) method to identify the relations between them. For NER, we explored deep learning-based approaches (ie, bidirectional long-short term memory with conditional random fields [BiLSTM-CRFs]) with various embeddings (ie, word embedding, character embedding [CE], and semantic-feature embedding) to investigate how different embeddings influence the performance. A rule-based method was implemented for RE and compared with a context-aware long-short term memory (LSTM) model. The methods were trained and evaluated using the 2018 n2c2 shared task data. Results: The experiments showed that the best model (BiLSTM-CRFs with pretrained word embeddings [PWE] and CE) achieved lenient micro F-scores of 0.921 for NER, 0.927 for RE, and 0.855 for the end-to-end system. NER, which relies on the pretrained word and semantic embeddings, performed better on most individual entity types, but NER with PWE and CE had the highest classification efficiency among the proposed approaches. Extracting relations using the rule-based method achieved higher accuracy than the context-aware LSTM for most relations. Interestingly, the LSTM model performed notably better in the reason-drug relations, the most challenging relation type. Conclusions: The proposed end-to-end system achieved encouraging results and demonstrated the feasibility of using deep learning methods to extract medication information from free-text data. UR - https://medinform.jmir.org/2021/5/e24678 UR - http://dx.doi.org/10.2196/24678 UR - http://www.ncbi.nlm.nih.gov/pubmed/33949962 ID - info:doi/10.2196/24678 ER - TY - JOUR AU - Chatterjee, Ayan AU - Prinz, Andreas AU - Gerdes, Martin AU - Martinez, Santiago PY - 2021/4/9 TI - An Automatic Ontology-Based Approach to Support Logical Representation of Observable and Measurable Data for Healthy Lifestyle Management: Proof-of-Concept Study JO - J Med Internet Res SP - e24656 VL - 23 IS - 4 KW - activity KW - nutrition KW - sensor KW - questionnaire KW - SSN KW - ontology KW - SNOMED CT KW - eCoach KW - personalized KW - recommendation KW - automated KW - CDSS KW - healthy lifestyle KW - interoperability KW - eHealth KW - goal setting KW - semantics KW - simulation KW - proposition N2 - Background: Lifestyle diseases, because of adverse health behavior, are the foremost cause of death worldwide. An eCoach system may encourage individuals to lead a healthy lifestyle with early health risk prediction, personalized recommendation generation, and goal evaluation. Such an eCoach system needs to collect and transform distributed heterogenous health and wellness data into meaningful information to train an artificially intelligent health risk prediction model. However, it may produce a data compatibility dilemma. Our proposed eHealth ontology can increase interoperability between different heterogeneous networks, provide situation awareness, help in data integration, and discover inferred knowledge. This ?proof-of-concept? study will help sensor, questionnaire, and interview data to be more organized for health risk prediction and personalized recommendation generation targeting obesity as a study case. Objective: The aim of this study is to develop an OWL-based ontology (UiA eHealth Ontology/UiAeHo) model to annotate personal, physiological, behavioral, and contextual data from heterogeneous sources (sensor, questionnaire, and interview), followed by structuring and standardizing of diverse descriptions to generate meaningful, practical, personalized, and contextual lifestyle recommendations based on the defined rules. Methods: We have developed a simulator to collect dummy personal, physiological, behavioral, and contextual data related to artificial participants involved in health monitoring. We have integrated the concepts of ?Semantic Sensor Network Ontology? and ?Systematized Nomenclature of Medicine?Clinical Terms? to develop our proposed eHealth ontology. The ontology has been created using Protégé (version 5.x). We have used the Java-based ?Jena Framework? (version 3.16) for building a semantic web application that includes resource description framework (RDF) application programming interface (API), OWL API, native tuple store (tuple database), and the SPARQL (Simple Protocol and RDF Query Language) query engine. The logical and structural consistency of the proposed ontology has been evaluated with the ?HermiT 1.4.3.x? ontology reasoner available in Protégé 5.x. Results: The proposed ontology has been implemented for the study case ?obesity.? However, it can be extended further to other lifestyle diseases. ?UiA eHealth Ontology? has been constructed using logical axioms, declaration axioms, classes, object properties, and data properties. The ontology can be visualized with ?Owl Viz,? and the formal representation has been used to infer a participant?s health status using the ?HermiT? reasoner. We have also developed a module for ontology verification that behaves like a rule-based decision support system to predict the probability for health risk, based on the evaluation of the results obtained from SPARQL queries. Furthermore, we discussed the potential lifestyle recommendation generation plan against adverse behavioral risks. Conclusions: This study has led to the creation of a meaningful, context-specific ontology to model massive, unintuitive, raw, unstructured observations for health and wellness data (eg, sensors, interviews, questionnaires) and to annotate them with semantic metadata to create a compact, intelligible abstraction for health risk predictions for individualized recommendation generation. UR - https://www.jmir.org/2021/4/e24656 UR - http://dx.doi.org/10.2196/24656 UR - http://www.ncbi.nlm.nih.gov/pubmed/33835031 ID - info:doi/10.2196/24656 ER - TY - JOUR AU - Park, Jimyung AU - You, Chan Seng AU - Jeong, Eugene AU - Weng, Chunhua AU - Park, Dongsu AU - Roh, Jin AU - Lee, Yun Dong AU - Cheong, Youn Jae AU - Choi, Wook Jin AU - Kang, Mira AU - Park, Woong Rae PY - 2021/3/30 TI - A Framework (SOCRATex) for Hierarchical Annotation of Unstructured Electronic Health Records and Integration Into a Standardized Medical Database: Development and Usability Study JO - JMIR Med Inform SP - e23983 VL - 9 IS - 3 KW - natural language processing KW - search engine KW - data curation KW - data management KW - common data model N2 - Background: Although electronic health records (EHRs) have been widely used in secondary assessments, clinical documents are relatively less utilized owing to the lack of standardized clinical text frameworks across different institutions. Objective: This study aimed to develop a framework for processing unstructured clinical documents of EHRs and integration with standardized structured data. Methods: We developed a framework known as Staged Optimization of Curation, Regularization, and Annotation of clinical text (SOCRATex). SOCRATex has the following four aspects: (1) extracting clinical notes for the target population and preprocessing the data, (2) defining the annotation schema with a hierarchical structure, (3) performing document-level hierarchical annotation using the annotation schema, and (4) indexing annotations for a search engine system. To test the usability of the proposed framework, proof-of-concept studies were performed on EHRs. We defined three distinctive patient groups and extracted their clinical documents (ie, pathology reports, radiology reports, and admission notes). The documents were annotated and integrated into the Observational Medical Outcomes Partnership (OMOP)-common data model (CDM) database. The annotations were used for creating Cox proportional hazard models with different settings of clinical analyses to measure (1) all-cause mortality, (2) thyroid cancer recurrence, and (3) 30-day hospital readmission. Results: Overall, 1055 clinical documents of 953 patients were extracted and annotated using the defined annotation schemas. The generated annotations were indexed into an unstructured textual data repository. Using the annotations of pathology reports, we identified that node metastasis and lymphovascular tumor invasion were associated with all-cause mortality among colon and rectum cancer patients (both P=.02). The other analyses involving measuring thyroid cancer recurrence using radiology reports and 30-day hospital readmission using admission notes in depressive disorder patients also showed results consistent with previous findings. Conclusions: We propose a framework for hierarchical annotation of textual data and integration into a standardized OMOP-CDM medical database. The proof-of-concept studies demonstrated that our framework can effectively process and integrate diverse clinical documents with standardized structured data for clinical research. UR - https://medinform.jmir.org/2021/3/e23983 UR - http://dx.doi.org/10.2196/23983 UR - http://www.ncbi.nlm.nih.gov/pubmed/33783361 ID - info:doi/10.2196/23983 ER - TY - JOUR AU - Ridgway, P. Jessica AU - Uvin, Arno AU - Schmitt, Jessica AU - Oliwa, Tomasz AU - Almirol, Ellen AU - Devlin, Samantha AU - Schneider, John PY - 2021/3/10 TI - Natural Language Processing of Clinical Notes to Identify Mental Illness and Substance Use Among People Living with HIV: Retrospective Cohort Study JO - JMIR Med Inform SP - e23456 VL - 9 IS - 3 KW - natural language processing KW - HIV KW - substance use KW - mental illness KW - electronic medical records N2 - Background: Mental illness and substance use are prevalent among people living with HIV and often lead to poor health outcomes. Electronic medical record (EMR) data are increasingly being utilized for HIV-related clinical research and care, but mental illness and substance use are often underdocumented in structured EMR fields. Natural language processing (NLP) of unstructured text of clinical notes in the EMR may more accurately identify mental illness and substance use among people living with HIV than structured EMR fields alone. Objective: The aim of this study was to utilize NLP of clinical notes to detect mental illness and substance use among people living with HIV and to determine how often these factors are documented in structured EMR fields. Methods: We collected both structured EMR data (diagnosis codes, social history, Problem List) as well as the unstructured text of clinical HIV care notes for adults living with HIV. We developed NLP algorithms to identify words and phrases associated with mental illness and substance use in the clinical notes. The algorithms were validated based on chart review. We compared numbers of patients with documentation of mental illness or substance use identified by structured EMR fields with those identified by the NLP algorithms. Results: The NLP algorithm for detecting mental illness had a positive predictive value (PPV) of 98% and a negative predictive value (NPV) of 98%. The NLP algorithm for detecting substance use had a PPV of 92% and an NPV of 98%. The NLP algorithm for mental illness identified 54.0% (420/778) of patients as having documentation of mental illness in the text of clinical notes. Among the patients with mental illness detected by NLP, 58.6% (246/420) had documentation of mental illness in at least one structured EMR field. Sixty-three patients had documentation of mental illness in structured EMR fields that was not detected by NLP of clinical notes. The NLP algorithm for substance use detected substance use in the text of clinical notes in 18.1% (141/778) of patients. Among patients with substance use detected by NLP, 73.8% (104/141) had documentation of substance use in at least one structured EMR field. Seventy-six patients had documentation of substance use in structured EMR fields that was not detected by NLP of clinical notes. Conclusions: Among patients in an urban HIV care clinic, NLP of clinical notes identified high rates of mental illness and substance use that were often not documented in structured EMR fields. This finding has important implications for epidemiologic research and clinical care for people living with HIV. UR - https://medinform.jmir.org/2021/3/e23456 UR - http://dx.doi.org/10.2196/23456 UR - http://www.ncbi.nlm.nih.gov/pubmed/33688848 ID - info:doi/10.2196/23456 ER - TY - JOUR AU - Yin, Zhijun AU - Liu, Yongtai AU - McCoy, B. Allison AU - Malin, A. Bradley AU - Sengstack, R. Patricia PY - 2021/3/4 TI - Contribution of Free-Text Comments to the Burden of Documentation: Assessment and Analysis of Vital Sign Comments in Flowsheets JO - J Med Internet Res SP - e22806 VL - 23 IS - 3 KW - electronic health system KW - documentation burden KW - flowsheets KW - content analysis KW - vital sign comments KW - free text N2 - Background: Documentation burden is a common problem with modern electronic health record (EHR) systems. To reduce this burden, various recording methods (eg, voice recorders or motion sensors) have been proposed. However, these solutions are in an early prototype phase and are unlikely to transition into practice in the near future. A more pragmatic alternative is to directly modify the implementation of the existing functionalities of an EHR system. Objective: This study aims to assess the nature of free-text comments entered into EHR flowsheets that supplement quantitative vital sign values and examine opportunities to simplify functionality and reduce documentation burden. Methods: We evaluated 209,055 vital sign comments in flowsheets that were generated in the Epic EHR system at the Vanderbilt University Medical Center in 2018. We applied topic modeling, as well as the natural language processing Clinical Language Annotation, Modeling, and Processing software system, to extract generally discussed topics and detailed medical terms (expressed as probability distribution) to investigate the stories communicated in these comments. Results: Our analysis showed that 63.33% (6053/9557) of the users who entered vital signs made at least one free-text comment in vital sign flowsheet entries. The user roles that were most likely to compose comments were registered nurse, technician, and licensed nurse. The most frequently identified topics were the notification of a result to health care providers (0.347), the context of a measurement (0.307), and an inability to obtain a vital sign (0.224). There were 4187 unique medical terms that were extracted from 46,029 (0.220) comments, including many symptom-related terms such as ?pain,? ?upset,? ?dizziness,? ?coughing,? ?anxiety,? ?distress,? and ?fever? and drug-related terms such as ?tylenol,? ?anesthesia,? ?cannula,? ?oxygen,? ?motrin,? ?rituxan,? and ?labetalol.? Conclusions: Considering that flowsheet comments are generally not displayed or automatically pulled into any clinical notes, our findings suggest that the flowsheet comment functionality can be simplified (eg, via structured response fields instead of a text input dialog) to reduce health care provider effort. Moreover, rich and clinically important medical terms such as medications and symptoms should be explicitly recorded in clinical notes for better visibility. UR - https://www.jmir.org/2021/3/e22806 UR - http://dx.doi.org/10.2196/22806 UR - http://www.ncbi.nlm.nih.gov/pubmed/33661128 ID - info:doi/10.2196/22806 ER - TY - JOUR AU - Parikh, Soham AU - Davoudi, Anahita AU - Yu, Shun AU - Giraldo, Carolina AU - Schriver, Emily AU - Mowery, Danielle PY - 2021/2/22 TI - Lexicon Development for COVID-19-related Concepts Using Open-source Word Embedding Sources: An Intrinsic and Extrinsic Evaluation JO - JMIR Med Inform SP - e21679 VL - 9 IS - 2 KW - natural language processing KW - word embedding KW - COVID-19 KW - intrinsic KW - open-source KW - computation KW - model KW - prediction KW - semantic KW - syntactic KW - pattern N2 - Background: Scientists are developing new computational methods and prediction models to better clinically understand COVID-19 prevalence, treatment efficacy, and patient outcomes. These efforts could be improved by leveraging documented COVID-19?related symptoms, findings, and disorders from clinical text sources in an electronic health record. Word embeddings can identify terms related to these clinical concepts from both the biomedical and nonbiomedical domains, and are being shared with the open-source community at large. However, it?s unclear how useful openly available word embeddings are for developing lexicons for COVID-19?related concepts. Objective: Given an initial lexicon of COVID-19?related terms, this study aims to characterize the returned terms by similarity across various open-source word embeddings and determine common semantic and syntactic patterns between the COVID-19 queried terms and returned terms specific to the word embedding source. Methods: We compared seven openly available word embedding sources. Using a series of COVID-19?related terms for associated symptoms, findings, and disorders, we conducted an interannotator agreement study to determine how accurately the most similar returned terms could be classified according to semantic types by three annotators. We conducted a qualitative study of COVID-19 queried terms and their returned terms to detect informative patterns for constructing lexicons. We demonstrated the utility of applying such learned synonyms to discharge summaries by reporting the proportion of patients identified by concept among three patient cohorts: pneumonia (n=6410), acute respiratory distress syndrome (n=8647), and COVID-19 (n=2397). Results: We observed high pairwise interannotator agreement (Cohen kappa) for symptoms (0.86-0.99), findings (0.93-0.99), and disorders (0.93-0.99). Word embedding sources generated based on characters tend to return more synonyms (mean count of 7.2 synonyms) compared to token-based embedding sources (mean counts range from 2.0 to 3.4). Word embedding sources queried using a qualifier term (eg, dry cough or muscle pain) more often returned qualifiers of the similar semantic type (eg, ?dry? returns consistency qualifiers like ?wet? and ?runny?) compared to a single term (eg, cough or pain) queries. A higher proportion of patients had documented fever (0.61-0.84), cough (0.41-0.55), shortness of breath (0.40-0.59), and hypoxia (0.51-0.56) retrieved than other clinical features. Terms for dry cough returned a higher proportion of patients with COVID-19 (0.07) than the pneumonia (0.05) and acute respiratory distress syndrome (0.03) populations. Conclusions: Word embeddings are valuable technology for learning related terms, including synonyms. When leveraging openly available word embedding sources, choices made for the construction of the word embeddings can significantly influence the words learned. UR - https://medinform.jmir.org/2021/2/e21679 UR - http://dx.doi.org/10.2196/21679 UR - http://www.ncbi.nlm.nih.gov/pubmed/33544689 ID - info:doi/10.2196/21679 ER - TY - JOUR AU - Kim, Taehyeong AU - Han, Won Sung AU - Kang, Minji AU - Lee, Ha Se AU - Kim, Jong-Ho AU - Joo, Joon Hyung AU - Sohn, Wook Jang PY - 2021/2/22 TI - Similarity-Based Unsupervised Spelling Correction Using BioWordVec: Development and Usability Study of Bacterial Culture and Antimicrobial Susceptibility Reports JO - JMIR Med Inform SP - e25530 VL - 9 IS - 2 KW - spelling correction KW - natural language processing KW - bacteria KW - electronic health record N2 - Background: Existing bacterial culture test results for infectious diseases are written in unrefined text, resulting in many problems, including typographical errors and stop words. Effective spelling correction processes are needed to ensure the accuracy and reliability of data for the study of infectious diseases, including medical terminology extraction. If a dictionary is established, spelling algorithms using edit distance are efficient. However, in the absence of a dictionary, traditional spelling correction algorithms that utilize only edit distances have limitations. Objective: In this research, we proposed a similarity-based spelling correction algorithm using pretrained word embedding with the BioWordVec technique. This method uses a character-level N-grams?based distributed representation through unsupervised learning rather than the existing rule-based method. In other words, we propose a framework that detects and corrects typographical errors when a dictionary is not in place. Methods: For detected typographical errors not mapped to Systematized Nomenclature of Medicine (SNOMED) clinical terms, a correction candidate group with high similarity considering the edit distance was generated using pretrained word embedding from the clinical database. From the embedding matrix in which the vocabulary is arranged in descending order according to frequency, a grid search was used to search for candidate groups of similar words. Thereafter, the correction candidate words were ranked in consideration of the frequency of the words, and the typographical errors were finally corrected according to the ranking. Results: Bacterial identification words were extracted from 27,544 bacterial culture and antimicrobial susceptibility reports, and 16 types of spelling errors and 914 misspelled words were found. The similarity-based spelling correction algorithm using BioWordVec proposed in this research corrected 12 types of typographical errors and showed very high performance in correcting 97.48% (based on F1 score) of all spelling errors. Conclusions: This tool corrected spelling errors effectively in the absence of a dictionary based on bacterial identification words in bacterial culture and antimicrobial susceptibility reports. This method will help build a high-quality refined database of vast text data for electronic health records. UR - https://medinform.jmir.org/2021/2/e25530 UR - http://dx.doi.org/10.2196/25530 UR - http://www.ncbi.nlm.nih.gov/pubmed/33616536 ID - info:doi/10.2196/25530 ER - TY - JOUR AU - Chen, Yen-Pin AU - Lo, Yuan-Hsun AU - Lai, Feipei AU - Huang, Chien-Hua PY - 2021/1/27 TI - Disease Concept-Embedding Based on the Self-Supervised Method for Medical Information Extraction from Electronic Health Records and Disease Retrieval: Algorithm Development and Validation Study JO - J Med Internet Res SP - e25113 VL - 23 IS - 1 KW - electronic health record KW - EHR KW - disease embedding KW - disease retrieval KW - emergency department KW - concept KW - extraction KW - deep learning KW - machine learning KW - natural language processing KW - NLP N2 - Background: The electronic health record (EHR) contains a wealth of medical information. An organized EHR can greatly help doctors treat patients. In some cases, only limited patient information is collected to help doctors make treatment decisions. Because EHRs can serve as a reference for this limited information, doctors? treatment capabilities can be enhanced. Natural language processing and deep learning methods can help organize and translate EHR information into medical knowledge and experience. Objective: In this study, we aimed to create a model to extract concept embeddings from EHRs for disease pattern retrieval and further classification tasks. Methods: We collected 1,040,989 emergency department visits from the National Taiwan University Hospital Integrated Medical Database and 305,897 samples from the National Hospital and Ambulatory Medical Care Survey Emergency Department data. After data cleansing and preprocessing, the data sets were divided into training, validation, and test sets. We proposed a Transformer-based model to embed EHRs and used Bidirectional Encoder Representations from Transformers (BERT) to extract features from free text and concatenate features with structural data as input to our proposed model. Then, Deep InfoMax (DIM) and Simple Contrastive Learning of Visual Representations (SimCLR) were used for the unsupervised embedding of the disease concept. The pretrained disease concept-embedding model, named EDisease, was further finetuned to adapt to the critical care outcome prediction task. We evaluated the performance of embedding using t-distributed stochastic neighbor embedding (t-SNE) to perform dimension reduction for visualization. The performance of the finetuned predictive model was evaluated against published models using the area under the receiver operating characteristic (AUROC). Results: The performance of our model on the outcome prediction had the highest AUROC of 0.876. In the ablation study, the use of a smaller data set or fewer unsupervised methods for pretraining deteriorated the prediction performance. The AUROCs were 0.857, 0.870, and 0.868 for the model without pretraining, the model pretrained by only SimCLR, and the model pretrained by only DIM, respectively. On the smaller finetuning set, the AUROC was 0.815 for the proposed model. Conclusions: Through contrastive learning methods, disease concepts can be embedded meaningfully. Moreover, these methods can be used for disease retrieval tasks to enhance clinical practice capabilities. The disease concept model is also suitable as a pretrained model for subsequent prediction tasks. UR - http://www.jmir.org/2021/1/e25113/ UR - http://dx.doi.org/10.2196/25113 UR - http://www.ncbi.nlm.nih.gov/pubmed/33502324 ID - info:doi/10.2196/25113 ER - TY - JOUR AU - Gaudet-Blavignac, Christophe AU - Foufi, Vasiliki AU - Bjelogrlic, Mina AU - Lovis, Christian PY - 2021/1/26 TI - Use of the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) for Processing Free Text in Health Care: Systematic Scoping Review JO - J Med Internet Res SP - e24594 VL - 23 IS - 1 KW - SNOMED CT KW - natural language processing KW - scoping review KW - terminology N2 - Background: Interoperability and secondary use of data is a challenge in health care. Specifically, the reuse of clinical free text remains an unresolved problem. The Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT) has become the universal language of health care and presents characteristics of a natural language. Its use to represent clinical free text could constitute a solution to improve interoperability. Objective: Although the use of SNOMED and SNOMED CT has already been reviewed, its specific use in processing and representing unstructured data such as clinical free text has not. This review aims to better understand SNOMED CT's use for representing free text in medicine. Methods: A scoping review was performed on the topic by searching MEDLINE, Embase, and Web of Science for publications featuring free-text processing and SNOMED CT. A recursive reference review was conducted to broaden the scope of research. The review covered the type of processed data, the targeted language, the goal of the terminology binding, the method used and, when appropriate, the specific software used. Results: In total, 76 publications were selected for an extensive study. The language targeted by publications was 91% (n=69) English. The most frequent types of documents for which the terminology was used are complementary exam reports (n=18, 24%) and narrative notes (n=16, 21%). Mapping to SNOMED CT was the final goal of the research in 21% (n=16) of publications and a part of the final goal in 33% (n=25). The main objectives of mapping are information extraction (n=44, 39%), feature in a classification task (n=26, 23%), and data normalization (n=23, 20%). The method used was rule-based in 70% (n=53) of publications, hybrid in 11% (n=8), and machine learning in 5% (n=4). In total, 12 different software packages were used to map text to SNOMED CT concepts, the most frequent being Medtex, Mayo Clinic Vocabulary Server, and Medical Text Extraction Reasoning and Mapping System. Full terminology was used in 64% (n=49) of publications, whereas only a subset was used in 30% (n=23) of publications. Postcoordination was proposed in 17% (n=13) of publications, and only 5% (n=4) of publications specifically mentioned the use of the compositional grammar. Conclusions: SNOMED CT has been largely used to represent free-text data, most frequently with rule-based approaches, in English. However, currently, there is no easy solution for mapping free text to this terminology and to perform automatic postcoordination. Most solutions conceive SNOMED CT as a simple terminology rather than as a compositional bag of ontologies. Since 2012, the number of publications on this subject per year has decreased. However, the need for formal semantic representation of free text in health care is high, and automatic encoding into a compositional ontology could be a solution. UR - http://www.jmir.org/2021/1/e24594/ UR - http://dx.doi.org/10.2196/24594 UR - http://www.ncbi.nlm.nih.gov/pubmed/33496673 ID - info:doi/10.2196/24594 ER - TY - JOUR AU - Kate, J. Rohit PY - 2021/1/14 TI - Clinical Term Normalization Using Learned Edit Patterns and Subconcept Matching: System Development and Evaluation JO - JMIR Med Inform SP - e23104 VL - 9 IS - 1 KW - clinical term normalization KW - edit distance KW - machine learning KW - natural language processing N2 - Background: Clinical terms mentioned in clinical text are often not in their standardized forms as listed in clinical terminologies because of linguistic and stylistic variations. However, many automated downstream applications require clinical terms mapped to their corresponding concepts in clinical terminologies, thus necessitating the task of clinical term normalization. Objective: In this paper, a system for clinical term normalization is presented that utilizes edit patterns to convert clinical terms into their normalized forms. Methods: The edit patterns are automatically learned from the Unified Medical Language System (UMLS) Metathesaurus as well as from the given training data. The edit patterns are generalized sequences of edits that are derived from edit distance computations. The edit patterns are both character based as well as word based and are learned separately for different semantic types. In addition to these edit patterns, the system also normalizes clinical terms through the subconcepts mentioned within them. Results: The system was evaluated as part of the 2019 n2c2 Track 3 shared task of clinical term normalization. It obtained 80.79% accuracy on the standard test data. This paper includes ablation studies to evaluate the contributions of different components of the system. A challenging part of the task was disambiguation when a clinical term could be normalized to multiple concepts. Conclusions: The learned edit patterns led the system to perform well on the normalization task. Given that the system is based on patterns, it is human interpretable and is also capable of giving insights about common variations of clinical terms mentioned in clinical text that are different from their standardized forms. UR - https://medinform.jmir.org/2021/1/e23104 UR - http://dx.doi.org/10.2196/23104 UR - http://www.ncbi.nlm.nih.gov/pubmed/33443483 ID - info:doi/10.2196/23104 ER - TY - JOUR AU - Rashidian, Sina AU - Abell-Hart, Kayley AU - Hajagos, Janos AU - Moffitt, Richard AU - Lingam, Veena AU - Garcia, Victor AU - Tsai, Chao-Wei AU - Wang, Fusheng AU - Dong, Xinyu AU - Sun, Siao AU - Deng, Jianyuan AU - Gupta, Rajarsi AU - Miller, Joshua AU - Saltz, Joel AU - Saltz, Mary PY - 2020/12/17 TI - Detecting Miscoded Diabetes Diagnosis Codes in Electronic Health Records for Quality Improvement: Temporal Deep Learning Approach JO - JMIR Med Inform SP - e22649 VL - 8 IS - 12 KW - electronic health records KW - diabetes KW - deep learning N2 - Background: Diabetes affects more than 30 million patients across the United States. With such a large disease burden, even a small error in classification can be significant. Currently billing codes, assigned at the time of a medical encounter, are the ?gold standard? reflecting the actual diseases present in an individual, and thus in aggregate reflect disease prevalence in the population. These codes are generated by highly trained coders and by health care providers but are not always accurate. Objective: This work provides a scalable deep learning methodology to more accurately classify individuals with diabetes across multiple health care systems. Methods: We leveraged a long short-term memory-dense neural network (LSTM-DNN) model to identify patients with or without diabetes using data from 5 acute care facilities with 187,187 patients and 275,407 encounters, incorporating data elements including laboratory test results, diagnostic/procedure codes, medications, demographic data, and admission information. Furthermore, a blinded physician panel reviewed discordant cases, providing an estimate of the total impact on the population. Results: When predicting the documented diagnosis of diabetes, our model achieved an 84% F1 score, 96% area under the curve?receiver operating characteristic curve, and 91% average precision on a heterogeneous data set from 5 distinct health facilities. However, in 81% of cases where the model disagreed with the documented phenotype, a blinded physician panel agreed with the model. Taken together, this suggests that 4.3% of our studied population have either missing or improper diabetes diagnosis. Conclusions: This study demonstrates that deep learning methods can improve clinical phenotyping even when patient data are noisy, sparse, and heterogeneous. UR - http://medinform.jmir.org/2020/12/e22649/ UR - http://dx.doi.org/10.2196/22649 UR - http://www.ncbi.nlm.nih.gov/pubmed/33331828 ID - info:doi/10.2196/22649 ER - TY - JOUR AU - Ryu, Borim AU - Yoon, Eunsil AU - Kim, Seok AU - Lee, Sejoon AU - Baek, Hyunyoung AU - Yi, Soyoung AU - Na, Young Hee AU - Kim, Ji-Won AU - Baek, Rong-Min AU - Hwang, Hee AU - Yoo, Sooyoung PY - 2020/12/9 TI - Transformation of Pathology Reports Into the Common Data Model With Oncology Module: Use Case for Colon Cancer JO - J Med Internet Res SP - e18526 VL - 22 IS - 12 KW - common data model KW - natural language processing KW - oncology module KW - colon cancer KW - electronic health record KW - oncology KW - pathology KW - clinical data N2 - Background: Common data models (CDMs) help standardize electronic health record data and facilitate outcome analysis for observational and longitudinal research. An analysis of pathology reports is required to establish fundamental information infrastructure for data-driven colon cancer research. The Observational Medical Outcomes Partnership (OMOP) CDM is used in distributed research networks for clinical data; however, it requires conversion of free text?based pathology reports into the CDM?s format. There are few use cases of representing cancer data in CDM. Objective: In this study, we aimed to construct a CDM database of colon cancer?related pathology with natural language processing (NLP) for a research platform that can utilize both clinical and omics data. The essential text entities from the pathology reports are extracted, standardized, and converted to the OMOP CDM format in order to utilize the pathology data in cancer research. Methods: We extracted clinical text entities, mapped them to the standard concepts in the Observational Health Data Sciences and Informatics vocabularies, and built databases and defined relations for the CDM tables. Major clinical entities were extracted through NLP on pathology reports of surgical specimens, immunohistochemical studies, and molecular studies of colon cancer patients at a tertiary general hospital in South Korea. Items were extracted from each report using regular expressions in Python. Unstructured data, such as text that does not have a pattern, were handled with expert advice by adding regular expression rules. Our own dictionary was used for normalization and standardization to deal with biomarker and gene names and other ungrammatical expressions. The extracted clinical and genetic information was mapped to the Logical Observation Identifiers Names and Codes databases and the Systematized Nomenclature of Medicine (SNOMED) standard terminologies recommended by the OMOP CDM. The database-table relationships were newly defined through SNOMED standard terminology concepts. The standardized data were inserted into the CDM tables. For evaluation, 100 reports were randomly selected and independently annotated by a medical informatics expert and a nurse. Results: We examined and standardized 1848 immunohistochemical study reports, 3890 molecular study reports, and 12,352 pathology reports of surgical specimens (from 2017 to 2018). The constructed and updated database contained the following extracted colorectal entities: (1) NOTE_NLP, (2) MEASUREMENT, (3) CONDITION_OCCURRENCE, (4) SPECIMEN, and (5) FACT_RELATIONSHIP of specimen with condition and measurement. Conclusions: This study aimed to prepare CDM data for a research platform to take advantage of all omics clinical and patient data at Seoul National University Bundang Hospital for colon cancer pathology. A more sophisticated preparation of the pathology data is needed for further research on cancer genomics, and various types of text narratives are the next target for additional research on the use of data in the CDM. UR - https://www.jmir.org/2020/12/e18526 UR - http://dx.doi.org/10.2196/18526 UR - http://www.ncbi.nlm.nih.gov/pubmed/33295294 ID - info:doi/10.2196/18526 ER - TY - JOUR AU - Lee, Jooyun AU - Park, Hyeoun-Ae AU - Park, Ki Seul AU - Song, Tae-Min PY - 2020/12/7 TI - Using Social Media Data to Understand Consumers' Information Needs and Emotions Regarding Cancer: Ontology-Based Data Analysis Study JO - J Med Internet Res SP - e18767 VL - 22 IS - 12 KW - social media KW - ontology KW - cancer KW - health information needs KW - cancer information KW - emotion N2 - Background: Analysis of posts on social media is effective in investigating health information needs for disease management and identifying people?s emotional status related to disease. An ontology is needed for semantic analysis of social media data. Objective: This study was performed to develop a cancer ontology with terminology containing consumer terms and to analyze social media data to identify health information needs and emotions related to cancer. Methods: A cancer ontology was developed using social media data, collected with a crawler, from online communities and blogs between January 1, 2014 and June 30, 2017 in South Korea. The relative frequencies of posts containing ontology concepts were counted and compared by cancer type. Results: The ontology had 9 superclasses, 213 class concepts, and 4061 synonyms. Ontology-driven natural language processing was performed on the text from 754,744 cancer-related posts. Colon, breast, stomach, cervical, lung, liver, pancreatic, and prostate cancer; brain tumors; and leukemia appeared most in these posts. At the superclass level, risk factor was the most frequent, followed by emotions, symptoms, treatments, and dealing with cancer. Conclusions: Information needs and emotions differed according to cancer type. The observations of this study could be used to provide tailored information to consumers according to cancer type and care process. Attention should be paid to provision of cancer-related information to not only patients but also their families and the general public seeking information on cancer. UR - http://www.jmir.org/2020/12/e18767/ UR - http://dx.doi.org/10.2196/18767 UR - http://www.ncbi.nlm.nih.gov/pubmed/33284127 ID - info:doi/10.2196/18767 ER - TY - JOUR AU - Luo, Lingyun AU - Feng, Jingtao AU - Yu, Huijun AU - Wang, Jiaolong PY - 2020/11/25 TI - Automatic Structuring of Ontology Terms Based on Lexical Granularity and Machine Learning: Algorithm Development and Validation JO - JMIR Med Inform SP - e22333 VL - 8 IS - 11 KW - ontology KW - automatic structuring KW - Foundational Model of Anatomy KW - lexical granularity KW - machine learning N2 - Background: As the manual creation and maintenance of biomedical ontologies are labor-intensive, automatic aids are desirable in the lifecycle of ontology development. Objective: Provided with a set of concept names in the Foundational Model of Anatomy (FMA), we propose an innovative method for automatically generating the taxonomy and the partonomy structures among them, respectively. Methods: Our approach comprises 2 main tasks: The first task is predicting the direct relation between 2 given concept names by utilizing word embedding methods and training 2 machine learning models, Convolutional Neural Networks (CNN) and Bidirectional Long Short-term Memory Networks (Bi-LSTM). The second task is the introduction of an original granularity-based method to identify the semantic structures among a group of given concept names by leveraging these trained models. Results: Results show that both CNN and Bi-LSTM perform well on the first task, with F1 measures above 0.91. For the second task, our approach achieves an average F1 measure of 0.79 on 100 case studies in the FMA using Bi-LSTM, which outperforms the primitive pairwise-based method. Conclusions: We have investigated an automatic way of predicting a hierarchical relationship between 2 concept names; based on this, we have further invented a methodology to structure a group of concept names automatically. This study is an initial investigation that will shed light on further work on the automatic creation and enrichment of biomedical ontologies. UR - http://medinform.jmir.org/2020/11/e22333/ UR - http://dx.doi.org/10.2196/22333 UR - http://www.ncbi.nlm.nih.gov/pubmed/33127601 ID - info:doi/10.2196/22333 ER - TY - JOUR AU - Cox, Steven AU - Ahalt, C. Stanley AU - Balhoff, James AU - Bizon, Chris AU - Fecho, Karamarie AU - Kebede, Yaphet AU - Morton, Kenneth AU - Tropsha, Alexander AU - Wang, Patrick AU - Xu, Hao PY - 2020/11/23 TI - Visualization Environment for Federated Knowledge Graphs: Development of an Interactive Biomedical Query Language and Web Application Interface JO - JMIR Med Inform SP - e17964 VL - 8 IS - 11 KW - knowledge graphs KW - clinical data KW - biomedical data KW - federation KW - ontologies KW - semantic harmonization KW - visualization KW - application programming interface KW - translational science KW - clinical practice N2 - Background: Efforts are underway to semantically integrate large biomedical knowledge graphs using common upper-level ontologies to federate graph-oriented application programming interfaces (APIs) to the data. However, federation poses several challenges, including query routing to appropriate knowledge sources, generation and evaluation of answer subsets, semantic merger of those answer subsets, and visualization and exploration of results. Objective: We aimed to develop an interactive environment for query, visualization, and deep exploration of federated knowledge graphs. Methods: We developed a biomedical query language and web application interphase?termed as Translator Query Language (TranQL)?to query semantically federated knowledge graphs and explore query results. TranQL uses the Biolink data model as an upper-level biomedical ontology and an API standard that has been adopted by the Biomedical Data Translator Consortium to specify a protocol for expressing a query as a graph of Biolink data elements compiled from statements in the TranQL query language. Queries are mapped to federated knowledge sources, and answers are merged into a knowledge graph, with mappings between the knowledge graph and specific elements of the query. The TranQL interactive web application includes a user interface to support user exploration of the federated knowledge graph. Results: We developed 2 real-world use cases to validate TranQL and address biomedical questions of relevance to translational science. The use cases posed questions that traversed 2 federated Translator API endpoints: Integrated Clinical and Environmental Exposures Service (ICEES) and Reasoning Over Biomedical Objects linked in Knowledge Oriented Pathways (ROBOKOP). ICEES provides open access to observational clinical and environmental data, and ROBOKOP provides access to linked biomedical entities, such as ?gene,? ?chemical substance,? and ?disease,? that are derived largely from curated public data sources. We successfully posed queries to TranQL that traversed these endpoints and retrieved answers that we visualized and evaluated. Conclusions: TranQL can be used to ask questions of relevance to translational science, rapidly obtain answers that require assertions from a federation of knowledge sources, and provide valuable insights for translational research and clinical practice. UR - http://medinform.jmir.org/2020/11/e17964/ UR - http://dx.doi.org/10.2196/17964 UR - http://www.ncbi.nlm.nih.gov/pubmed/33226347 ID - info:doi/10.2196/17964 ER - TY - JOUR AU - de Lusignan, Simon AU - Liyanage, Harshana AU - McGagh, Dylan AU - Jani, Dinesh Bhautesh AU - Bauwens, Jorgen AU - Byford, Rachel AU - Evans, Dai AU - Fahey, Tom AU - Greenhalgh, Trisha AU - Jones, Nicholas AU - Mair, S. Frances AU - Okusi, Cecilia AU - Parimalanathan, Vaishnavi AU - Pell, P. Jill AU - Sherlock, Julian AU - Tamburis, Oscar AU - Tripathy, Manasa AU - Ferreira, Filipa AU - Williams, John AU - Hobbs, Richard F. D. PY - 2020/11/17 TI - COVID-19 Surveillance in a Primary Care Sentinel Network: In-Pandemic Development of an Application Ontology JO - JMIR Public Health Surveill SP - e21434 VL - 6 IS - 4 KW - COVID-19 KW - medical informatics KW - sentinel surveillance N2 - Background: Creating an ontology for COVID-19 surveillance should help ensure transparency and consistency. Ontologies formalize conceptualizations at either the domain or application level. Application ontologies cross domains and are specified through testable use cases. Our use case was an extension of the role of the Oxford Royal College of General Practitioners (RCGP) Research and Surveillance Centre (RSC) to monitor the current pandemic and become an in-pandemic research platform. Objective: This study aimed to develop an application ontology for COVID-19 that can be deployed across the various use-case domains of the RCGP RSC research and surveillance activities. Methods: We described our domain-specific use case. The actor was the RCGP RSC sentinel network, the system was the course of the COVID-19 pandemic, and the outcomes were the spread and effect of mitigation measures. We used our established 3-step method to develop the ontology, separating ontological concept development from code mapping and data extract validation. We developed a coding system?independent COVID-19 case identification algorithm. As there were no gold-standard pandemic surveillance ontologies, we conducted a rapid Delphi consensus exercise through the International Medical Informatics Association Primary Health Care Informatics working group and extended networks. Results: Our use-case domains included primary care, public health, virology, clinical research, and clinical informatics. Our ontology supported (1) case identification, microbiological sampling, and health outcomes at an individual practice and at the national level; (2) feedback through a dashboard; (3) a national observatory; (4) regular updates for Public Health England; and (5) transformation of a sentinel network into a trial platform. We have identified a total of 19,115 people with a definite COVID-19 status, 5226 probable cases, and 74,293 people with possible COVID-19, within the RCGP RSC network (N=5,370,225). Conclusions: The underpinning structure of our ontological approach has coped with multiple clinical coding challenges. At a time when there is uncertainty about international comparisons, clarity about the basis on which case definitions and outcomes are made from routine data is essential. UR - http://publichealth.jmir.org/2020/4/e21434/ UR - http://dx.doi.org/10.2196/21434 UR - http://www.ncbi.nlm.nih.gov/pubmed/33112762 ID - info:doi/10.2196/21434 ER - TY - JOUR AU - Carlson, A. Luke AU - Jeffery, M. Molly AU - Fu, Sunyang AU - He, Huan AU - McCoy, G. Rozalina AU - Wang, Yanshan AU - Hooten, Michael William AU - St Sauver, Jennifer AU - Liu, Hongfang AU - Fan, Jungwei PY - 2020/11/16 TI - Characterizing Chronic Pain Episodes in Clinical Text at Two Health Care Systems: Comprehensive Annotation and Corpus Analysis JO - JMIR Med Inform SP - e18659 VL - 8 IS - 11 KW - chronic pain KW - guideline development KW - knowledge representation KW - corpus annotation KW - content analysis N2 - Background: Chronic pain affects more than 20% of adults in the United States and is associated with substantial physical, mental, and social burden. Clinical text contains rich information about chronic pain, but no systematic appraisal has been performed to assess the electronic health record (EHR) narratives for these patients. A formal content analysis of the unstructured EHR data can inform clinical practice and research in chronic pain. Objective: We characterized individual episodes of chronic pain by annotating and analyzing EHR notes for a stratified cohort of adults with known chronic pain. Methods: We used the Rochester Epidemiology Project infrastructure to screen all residents of Olmsted County, Minnesota, for evidence of chronic pain, between January 1, 2005, and September 30, 2015. Diagnosis codes were used to assemble a cohort of 6586 chronic pain patients; people with cancer were excluded. The records of an age- and sex-stratified random sample of 62 patients from the cohort were annotated using an iteratively developed guideline. The annotated concepts included date, location, severity, causes, effects on quality of life, diagnostic procedures, medications, and other treatment modalities. Results: A total of 94 chronic pain episodes from 62 distinct patients were identified by reviewing 3272 clinical notes. Documentation was written by clinicians across a wide spectrum of specialties. Most patients (40/62, 65%) had 1 pain episode during the study period. Interannotator agreement ranged from 0.78 to 1.00 across the annotated concepts. Some pain-related concepts (eg, body location) had 100% (94/94) coverage among all the episodes, while others had moderate coverage (eg, effects on quality of life) (55/94, 59%). Back pain and leg pain were the most common types of chronic pain in the annotated cohort. Musculoskeletal issues like arthritis were annotated as the most common causes. Opioids were the most commonly captured medication, while physical and occupational therapies were the most common nonpharmacological treatments. Conclusions: We systematically annotated chronic pain episodes in clinical text. The rich content analysis results revealed complexity of the chronic pain episodes and of their management, as well as the challenges in extracting pertinent information, even for humans. Despite the pilot study nature of the work, the annotation guideline and corpus should be able to serve as informative references for other institutions with shared interest in chronic pain research using EHRs. UR - http://medinform.jmir.org/2020/11/e18659/ UR - http://dx.doi.org/10.2196/18659 UR - http://www.ncbi.nlm.nih.gov/pubmed/33108311 ID - info:doi/10.2196/18659 ER - TY - JOUR AU - Lau, Francis AU - Antonio, Marcy AU - Davison, Kelly AU - Queen, Roz AU - Bryski, Katie PY - 2020/11/11 TI - An Environmental Scan of Sex and Gender in Electronic Health Records: Analysis of Public Information Sources JO - J Med Internet Res SP - e20050 VL - 22 IS - 11 KW - sex KW - gender KW - electronic health records KW - standards KW - transgender persons N2 - Background: Historically, the terms sex and gender have been used interchangeably as a binary attribute to describe a person as male or female, even though there is growing recognition that sex and gender are distinct concepts. The lack of sex and gender delineation in electronic health records (EHRs) may be perpetuating the inequities experienced by the transgender and gender nonbinary (TGNB) populations. Objective: This study aims to conduct an environmental scan to understand how sex and gender are defined and implemented in existing Canadian EHRs and current international health information standards. Methods: We examined public information sources on sex and gender definitions in existing Canadian EHRs and international standards communities. Definitions refer to data element names, code systems, and value sets in the descriptions of EHRs and standards. The study was built on an earlier environment scan by Canada Health Infoway, supplemented with sex and gender definitions from international standards communities. For the analysis, we examined the definitions for clarity, consistency, and accuracy. We also received feedback from a virtual community interested in sex-gender EHR issues. Results: The information sources consisted of public website descriptions of 52 databases and 55 data standards from 12 Canadian entities and 10 standards communities. There are variations in the definition and implementation of sex and gender in Canadian EHRs and international health information standards. There is a lack of clarity in some sex and gender concepts. There is inconsistency in the data element names, code systems, and value sets used to represent sex and gender concepts across EHRs. The appropriateness and adequacy of some value options are questioned as our societal understanding of sexual health evolves. Outdated value options raise concerns about current EHRs supporting the provision of culturally competent, safe, and affirmative health care. The limited options also perpetuate the inequities faced by the TGNB populations. The expanded sex and gender definitions from leading Canadian organizations and international standards communities have brought challenges in how to migrate these definitions into existing EHRs. We proposed 6 high-level actions, which are to articulate the need for this work, reach consensus on sex and gender concepts, reach consensus on expanded sex and gender definitions in EHRs, develop a coordinated action plan, embrace EHR change from socio-organizational and technical aspects to ensure success, and demonstrate the benefits in tangible terms. Conclusions: There are variations in sex and gender concepts across Canadian EHRs and the health information standards that support them. Although there are efforts to modernize sex and gender concept definitions, we need decisive and coordinated actions to ensure clarity, consistency, and competency in the definition and implementation of sex and gender concepts in EHRs. This work has implications for addressing the inequities of TGNB populations in Canada. UR - http://www.jmir.org/2020/11/e20050/ UR - http://dx.doi.org/10.2196/20050 UR - http://www.ncbi.nlm.nih.gov/pubmed/33174858 ID - info:doi/10.2196/20050 ER - TY - JOUR AU - Kang, Hongyu AU - Li, Jiao AU - Wu, Meng AU - Shen, Liu AU - Hou, Li PY - 2020/10/21 TI - Building a Pharmacogenomics Knowledge Model Toward Precision Medicine: Case Study in Melanoma JO - JMIR Med Inform SP - e20291 VL - 8 IS - 10 KW - pharmacogenomics KW - knowledge model KW - BERT?CRF model KW - named entity recognition KW - melanoma N2 - Background: Many drugs do not work the same way for everyone owing to distinctions in their genes. Pharmacogenomics (PGx) aims to understand how genetic variants influence drug efficacy and toxicity. It is often considered one of the most actionable areas of the personalized medicine paradigm. However, little prior work has included in-depth explorations and descriptions of drug usage, dosage adjustment, and so on. Objective: We present a pharmacogenomics knowledge model to discover the hidden relationships between PGx entities such as drugs, genes, and diseases, especially details in precise medication. Methods: PGx open data such as DrugBank and RxNorm were integrated in this study, as well as drug labels published by the US Food and Drug Administration. We annotated 190 drug labels manually for entities and relationships. Based on the annotation results, we trained 3 different natural language processing models to complete entity recognition. Finally, the pharmacogenomics knowledge model was described in detail. Results: In entity recognition tasks, the Bidirectional Encoder Representations from Transformers?conditional random field model achieved better performance with micro-F1 score of 85.12%. The pharmacogenomics knowledge model in our study included 5 semantic types: drug, gene, disease, precise medication (population, daily dose, dose form, frequency, etc), and adverse reaction. Meanwhile, 26 semantic relationships were defined in detail. Taking melanoma caused by a BRAF gene mutation into consideration, the pharmacogenomics knowledge model covered 7 related drugs and 4846 triples were established in this case. All the corpora, relationship definitions, and triples were made publically available. Conclusions: We highlighted the pharmacogenomics knowledge model as a scalable framework for clinicians and clinical pharmacists to adjust drug dosage according to patient-specific genetic variation, and for pharmaceutical researchers to develop new drugs. In the future, a series of other antitumor drugs and automatic relation extractions will be taken into consideration to further enhance our framework with more PGx linked data. UR - http://medinform.jmir.org/2020/10/e20291/ UR - http://dx.doi.org/10.2196/20291 UR - http://www.ncbi.nlm.nih.gov/pubmed/33084582 ID - info:doi/10.2196/20291 ER - TY - JOUR AU - Delvaux, Nicolas AU - Vaes, Bert AU - Aertgeerts, Bert AU - Van de Velde, Stijn AU - Vander Stichele, Robert AU - Nyberg, Peter AU - Vermandere, Mieke PY - 2020/10/21 TI - Coding Systems for Clinical Decision Support: Theoretical and Real-World Comparative Analysis JO - JMIR Form Res SP - e16094 VL - 4 IS - 10 KW - clinical decision support systems KW - clinical coding KW - medical informatics KW - electronic health records N2 - Background: Effective clinical decision support systems require accurate translation of practice recommendations into machine-readable artifacts; developing code sets that represent clinical concepts are an important step in this process. Many clinical coding systems are currently used in electronic health records, and it is unclear whether all of these systems are capable of efficiently representing the clinical concepts required in executing clinical decision support systems. Objective: The aim of this study was to evaluate which clinical coding systems are capable of efficiently representing clinical concepts that are necessary for translating artifacts into executable code for clinical decision support systems. Methods: Two methods were used to evaluate a set of clinical coding systems. In a theoretical approach, we extracted all the clinical concepts from 3 preventive care recommendations and constructed a series of code sets containing codes from a single clinical coding system. In a practical approach using data from a real-world setting, we studied the content of 1890 code sets used in an internationally available clinical decision support system and compared the usage of various clinical coding systems. Results: SNOMED CT and ICD-10 (International Classification of Diseases, Tenth Revision) proved to be the most accurate clinical coding systems for most concepts in our theoretical evaluation. In our practical evaluation, we found that International Classification of Diseases (Tenth Revision) was most often used to construct code sets. Some coding systems were very accurate in representing specific types of clinical concepts, for example, LOINC (Logical Observation Identifiers Names and Codes) for investigation results and ATC (Anatomical Therapeutic Chemical Classification) for drugs. Conclusions: No single coding system seems to fulfill all the needs for representing clinical concepts for clinical decision support systems. Comprehensiveness of the coding systems seems to be offset by complexity and forms a barrier to usability for code set construction. Clinical vocabularies mapped to multiple clinical coding systems could facilitate clinical code set construction. UR - http://formative.jmir.org/2020/10/e16094/ UR - http://dx.doi.org/10.2196/16094 UR - http://www.ncbi.nlm.nih.gov/pubmed/33084593 ID - info:doi/10.2196/16094 ER - TY - JOUR AU - Zhu, Qian AU - Nguyen, Dac-Trung AU - Alyea, Gioconda AU - Hanson, Karen AU - Sid, Eric AU - Pariser, Anne PY - 2020/10/2 TI - Phenotypically Similar Rare Disease Identification from an Integrative Knowledge Graph for Data Harmonization: Preliminary Study JO - JMIR Med Inform SP - e18395 VL - 8 IS - 10 KW - GARD KW - rare diseases KW - phenotypical similarity KW - data harmonization N2 - Background: Although many efforts have been made to develop comprehensive disease resources that capture rare disease information for the purpose of clinical decision making and education, there is no standardized protocol for defining and harmonizing rare diseases across multiple resources. This introduces data redundancy and inconsistency that may ultimately increase confusion and difficulty for the wide use of these resources. To overcome such encumbrances, we report our preliminary study to identify phenotypical similarity among genetic and rare diseases (GARD) that are presenting similar clinical manifestations, and support further data harmonization. Objective: To support rare disease data harmonization, we aim to systematically identify phenotypically similar GARD diseases from a disease-oriented integrative knowledge graph and determine their similarity types. Methods: We identified phenotypically similar GARD diseases programmatically with 2 methods: (1) We measured disease similarity by comparing disease mappings between GARD and other rare disease resources, incorporating manual assessment; 2) we derived clinical manifestations presenting among sibling diseases from disease classifications and prioritized the identified similar diseases based on their phenotypes and genotypes. Results: For disease similarity comparison, approximately 87% (341/392) identified, phenotypically similar disease pairs were validated; 80% (271/392) of these disease pairs were accurately identified as phenotypically similar based on similarity score. The evaluation result shows a high precision (94%) and a satisfactory quality (86% F measure). By deriving phenotypical similarity from Monarch Disease Ontology (MONDO) and Orphanet disease classification trees, we identified a total of 360 disease pairs with at least 1 shared clinical phenotype and gene, which were applied for prioritizing clinical relevance. A total of 662 phenotypically similar disease pairs were identified and will be applied for GARD data harmonization. Conclusions: We successfully identified phenotypically similar rare diseases among the GARD diseases via 2 approaches, disease mapping comparison and phenotypical similarity derivation from disease classification systems. The results will not only direct GARD data harmonization in expanding translational science research but will also accelerate data transparency and consistency across different disease resources and terminologies, helping to build a robust and up-to-date knowledge resource on rare diseases. UR - https://medinform.jmir.org/2020/10/e18395 UR - http://dx.doi.org/10.2196/18395 UR - http://www.ncbi.nlm.nih.gov/pubmed/33006565 ID - info:doi/10.2196/18395 ER - TY - JOUR AU - Li, Yongbin AU - Wang, Xiaohua AU - Hui, Linhu AU - Zou, Liping AU - Li, Hongjin AU - Xu, Luo AU - Liu, Weihai PY - 2020/9/4 TI - Chinese Clinical Named Entity Recognition in Electronic Medical Records: Development of a Lattice Long Short-Term Memory Model With Contextualized Character Representations JO - JMIR Med Inform SP - e19848 VL - 8 IS - 9 KW - clinical named entity recognition KW - ELMo KW - lattice LSTM KW - deep learning KW - neural network KW - sequence tagging N2 - Background: Clinical named entity recognition (CNER), whose goal is to automatically identify clinical entities in electronic medical records (EMRs), is an important research direction of clinical text data mining and information extraction. The promotion of CNER can provide support for clinical decision making and medical knowledge base construction, which could then improve overall medical quality. Compared with English CNER, and due to the complexity of Chinese word segmentation and grammar, Chinese CNER was implemented later and is more challenging. Objective: With the development of distributed representation and deep learning, a series of models have been applied in Chinese CNER. Different from the English version, Chinese CNER is mainly divided into character-based and word-based methods that cannot make comprehensive use of EMR information and cannot solve the problem of ambiguity in word representation. Methods: In this paper, we propose a lattice long short-term memory (LSTM) model combined with a variant contextualized character representation and a conditional random field (CRF) layer for Chinese CNER: the Embeddings from Language Models (ELMo)-lattice-LSTM-CRF model. The lattice LSTM model can effectively utilize the information from characters and words in Chinese EMRs; in addition, the variant ELMo model uses Chinese characters as input instead of the character-encoding layer of the ELMo model, so as to learn domain-specific contextualized character embeddings. Results: We evaluated our method using two Chinese CNER datasets from the China Conference on Knowledge Graph and Semantic Computing (CCKS): the CCKS-2017 CNER dataset and the CCKS-2019 CNER dataset. We obtained F1 scores of 90.13% and 85.02% on the test sets of these two datasets, respectively. Conclusions: Our results show that our proposed method is effective in Chinese CNER. In addition, the results of our experiments show that variant contextualized character representations can significantly improve the performance of the model. UR - http://medinform.jmir.org/2020/9/e19848/ UR - http://dx.doi.org/10.2196/19848 UR - http://www.ncbi.nlm.nih.gov/pubmed/32885786 ID - info:doi/10.2196/19848 ER - TY - JOUR AU - Nasralah, Tareq AU - El-Gayar, Omar AU - Wang, Yong PY - 2020/8/13 TI - Social Media Text Mining Framework for Drug Abuse: Development and Validation Study With an Opioid Crisis Case Analysis JO - J Med Internet Res SP - e18350 VL - 22 IS - 8 KW - drug abuse KW - social media KW - infodemiology KW - infoveillance KW - text mining KW - opioid crisis N2 - Background: Social media are considered promising and viable sources of data for gaining insights into various disease conditions and patients? attitudes, behaviors, and medications. They can be used to recognize communication and behavioral themes of problematic use of prescription drugs. However, mining and analyzing social media data have challenges and limitations related to topic deduction and data quality. As a result, we need a structured approach to analyze social media content related to drug abuse in a manner that can mitigate the challenges and limitations surrounding the use of such data. Objective: This study aimed to develop and evaluate a framework for mining and analyzing social media content related to drug abuse. The framework is designed to mitigate challenges and limitations related to topic deduction and data quality in social media data analytics for drug abuse. Methods: The proposed framework started with defining different terms related to the keywords, categories, and characteristics of the topic of interest. We then used the Crimson Hexagon platform to collect data based on a search query informed by a drug abuse ontology developed using the identified terms. We subsequently preprocessed the data and examined the quality using an evaluation matrix. Finally, a suitable data analysis approach could be used to analyze the collected data. Results: The framework was evaluated using the opioid epidemic as a drug abuse case analysis. We demonstrated the applicability of the proposed framework to identify public concerns toward the opioid epidemic and the most discussed topics on social media related to opioids. The results from the case analysis showed that the framework could improve the discovery and identification of topics in social media domains characterized by a plethora of highly diverse terms and lack of a commonly available dictionary or language by the community, such as in the case of opioid and drug abuse. Conclusions: The proposed framework addressed the challenges related to topic detection and data quality. We demonstrated the applicability of the proposed framework to identify the common concerns toward the opioid epidemic and the most discussed topics on social media related to opioids. UR - https://www.jmir.org/2020/8/e18350 UR - http://dx.doi.org/10.2196/18350 UR - http://www.ncbi.nlm.nih.gov/pubmed/32788147 ID - info:doi/10.2196/18350 ER - TY - JOUR AU - Bao, Hongchang AU - Baker, O. Christopher J. AU - Adisesh, Anil PY - 2020/8/5 TI - Occupation Coding of Job Titles: Iterative Development of an Automated Coding Algorithm for the Canadian National Occupation Classification (ACA-NOC) JO - JMIR Form Res SP - e16422 VL - 4 IS - 8 KW - occupation coding KW - automated coding KW - occupational health KW - job title N2 - Background: In many research studies, the identification of social determinants is an important activity, in particular, information about occupations is frequently added to existing patient data. Such information is usually solicited during interviews with open-ended questions such as ?What is your job?? and ?What industry sector do you work in?? Before being able to use this information for further analysis, the responses need to be categorized using a coding system, such as the Canadian National Occupational Classification (NOC). Manual coding is the usual method, which is a time-consuming and error-prone activity, suitable for automation. Objective: This study aims to facilitate automated coding by introducing a rigorous algorithm that will be able to identify the NOC (2016) codes using only job title and industry information as input. Using manually coded data sets, we sought to benchmark and iteratively improve the performance of the algorithm. Methods: We developed the ACA-NOC algorithm based on the NOC (2016), which allowed users to match NOC codes with job and industry titles. We employed several different search strategies in the ACA-NOC algorithm to find the best match, including exact search, minor exact search, like search, near (same order) search, near (different order) search, any search, and weak match search. In addition, a filtering step based on the hierarchical structure of the NOC data was applied to the algorithm to select the best matching codes. Results: The ACA-NOC was applied to over 500 manually coded job and industry titles. The accuracy rate at the four-digit NOC code level was 58.7% (332/566) and improved when broader job categories were considered (65.0% at the three-digit NOC code level, 72.3% at the two-digit NOC code level, and 81.6% at the one-digit NOC code level). Conclusions: The ACA-NOC is a rigorous algorithm for automatically coding the Canadian NOC system and has been evaluated using real-world data. It allows researchers to code moderate-sized data sets with occupation in a timely and cost-efficient manner such that further analytics are possible. Initial assessments indicate that it has state-of-the-art performance and is readily extensible upon further benchmarking on larger data sets. UR - https://formative.jmir.org/2020/8/e16422 UR - http://dx.doi.org/10.2196/16422 UR - http://www.ncbi.nlm.nih.gov/pubmed/32755893 ID - info:doi/10.2196/16422 ER - TY - JOUR AU - Pan, Xiaoyi AU - Chen, Boyu AU - Weng, Heng AU - Gong, Yongyi AU - Qu, Yingying PY - 2020/7/27 TI - Temporal Expression Classification and Normalization From Chinese Narrative Clinical Texts: Pattern Learning Approach JO - JMIR Med Inform SP - e17652 VL - 8 IS - 7 KW - Temporal expression extraction KW - Temporal expression normalization KW - Machine learning KW - Heuristic rule KW - Pattern learning KW - Clinical text N2 - Background: Temporal information frequently exists in the representation of the disease progress, prescription, medication, surgery progress, or discharge summary in narrative clinical text. The accurate extraction and normalization of temporal expressions can positively boost the analysis and understanding of narrative clinical texts to promote clinical research and practice. Objective: The goal of the study was to propose a novel approach for extracting and normalizing temporal expressions from Chinese narrative clinical text. Methods: TNorm, a rule-based and pattern learning-based approach, has been developed for automatic temporal expression extraction and normalization from unstructured Chinese clinical text data. TNorm consists of three stages: extraction, classification, and normalization. It applies a set of heuristic rules and automatically generated patterns for temporal expression identification and extraction of clinical texts. Then, it collects the features of extracted temporal expressions for temporal type prediction and classification by using machine learning algorithms. Finally, the features are combined with the rule-based and a pattern learning-based approach to normalize the extracted temporal expressions. Results: The evaluation dataset is a set of narrative clinical texts in Chinese containing 1459 discharge summaries of a domestic Grade A Class 3 hospital. The results show that TNorm, combined with temporal expressions extraction and temporal types prediction, achieves a precision of 0.8491, a recall of 0.8328, and a F1 score of 0.8409 in temporal expressions normalization. Conclusions: This study illustrates an automatic approach, TNorm, that extracts and normalizes temporal expression from Chinese narrative clinical texts. TNorm was evaluated on the basis of discharge summary data, and results demonstrate its effectiveness on temporal expression normalization. UR - https://medinform.jmir.org/2020/7/e17652 UR - http://dx.doi.org/10.2196/17652 UR - http://www.ncbi.nlm.nih.gov/pubmed/32716307 ID - info:doi/10.2196/17652 ER - TY - JOUR AU - Mangin, Dee AU - Lawson, Jennifer AU - Adamczyk, Krzysztof AU - Guenter, Dale PY - 2020/7/27 TI - Embedding ?Smart? Disease Coding Within Routine Electronic Medical Record Workflow: Prospective Single-Arm Trial JO - JMIR Med Inform SP - e16764 VL - 8 IS - 7 KW - chronic disease management KW - comorbidity KW - problem list KW - disease coding KW - disease registry KW - data improvement KW - electronic medical record KW - electronic health record KW - practice-based research network KW - population health KW - primary care KW - family medicine N2 - Background: Electronic medical record (EMR) chronic disease measurement can help direct primary care prevention and treatment strategies and plan health services resource management. Incomplete data and poor consistency of coded disease values within EMR problem lists are widespread issues that limit primary and secondary uses of these data. These issues were shared by the McMaster University Sentinel and Information Collaboration (MUSIC), a primary care practice-based research network (PBRN) located in Hamilton, Ontario, Canada. Objective: We sought to develop and evaluate the effectiveness of new EMR interface tools aimed at improving the quantity and the consistency of disease codes recorded within the disease registry across the MUSIC PBRN. Methods: We used a single-arm prospective trial design with preintervention and postintervention data analysis to assess the effect of the intervention on disease recording volume and quality. The MUSIC network holds data on over 75,080 patients, 37,212 currently rostered. There were 4 MUSIC network clinician champions involved in gap analysis of the disease coding process and in the iterative design of new interface tools. We leveraged terminology standards and factored EMR workflow and usability into a new interface solution that aimed to optimize code selection volume and quality while minimizing physician time burden. The intervention was integrated as part of usual clinical workflow during routine billing activities. Results: After implementation of the new interface (June 25, 2017), we assessed the disease registry codes at 3 and 6 months (intervention period) to compare their volume and quality to preintervention levels (baseline period). A total of 17,496 International Classification of Diseases, 9th Revision (ICD9) code values were recorded in the disease registry during the 11.5-year (2006 to mid-2017) baseline period. A large gain in disease recording occurred in the intervention period (8516/17,496, 48.67% over baseline), resulting in a total of 26,774 codes. The coding rate increased by a factor of 11.2, averaging 1419 codes per month over the baseline average rate of 127 codes per month. The proportion of preferred ICD9 codes increased by 17.03% in the intervention period (11,007/17,496, 62.91% vs 7417/9278, 79.94%; ?21=819.4; P<.001). A total of 45.03% (4178/9278) of disease codes were entered by way of the new screen prompt tools, with significant increases between quarters (Jul-Sep: 2507/6140, 40.83% vs Oct-Dec: 1671/3148, 53.08%; ?21=126.2; P<.001). Conclusions: The introduction of clinician co-designed, workflow-embedded disease coding tools is a very effective solution to the issues of poor disease coding and quality in EMRs. The substantial effectiveness in a routine care environment demonstrates usability, and the intervention detail described here should be generalizable to any setting. Significant improvements in problem list coding within primary care EMRs can be realized with minimal disruption to routine clinical workflow. UR - http://medinform.jmir.org/2020/7/e16764/ UR - http://dx.doi.org/10.2196/16764 UR - http://www.ncbi.nlm.nih.gov/pubmed/32716304 ID - info:doi/10.2196/16764 ER - TY - JOUR AU - Li, Xiaoying AU - Lin, Xin AU - Ren, Huiling AU - Guo, Jinjing PY - 2020/7/20 TI - Ontological Organization and Bioinformatic Analysis of Adverse Drug Reactions From Package Inserts: Development and Usability Study JO - J Med Internet Res SP - e20443 VL - 22 IS - 7 KW - ontology KW - adverse drug reactions KW - package inserts KW - information retrieval KW - natural language processing KW - bioinformatics KW - drug KW - adverse events KW - machine-understandable knowledge KW - clinical applications N2 - Background: Licensed drugs may cause unexpected adverse reactions in patients, resulting in morbidity, risk of mortality, therapy disruptions, and prolonged hospital stays. Officially approved drug package inserts list the adverse reactions identified from randomized controlled clinical trials with high evidence levels and worldwide postmarketing surveillance. Formal representation of the adverse drug reaction (ADR) enclosed in semistructured package inserts will enable deep recognition of side effects and rational drug use, substantially reduce morbidity, and decrease societal costs. Objective: This paper aims to present an ontological organization of traceable ADR information extracted from licensed package inserts. In addition, it will provide machine-understandable knowledge for bioinformatics analysis, semantic retrieval, and intelligent clinical applications. Methods: Based on the essential content of package inserts, a generic ADR ontology model is proposed from two dimensions (and nine subdimensions), covering the ADR information and medication instructions. This is followed by a customized natural language processing method programmed with Python to retrieve the relevant information enclosed in package inserts. After the biocuration and identification of retrieved data from the package insert, an ADR ontology is automatically built for further bioinformatic analysis. Results: We collected 165 package inserts of quinolone drugs from the National Medical Products Administration and other drug databases in China, and built a specialized ADR ontology containing 2879 classes and 15,711 semantic relations. For each quinolone drug, the reported ADR information and medication instructions have been logically represented and formally organized in an ADR ontology. To demonstrate its usage, the source data were further bioinformatically analyzed. For example, the number of drug-ADR triples and major ADRs associated with each active ingredient were recorded. The 10 ADRs most frequently observed among quinolones were identified and categorized based on the 18 categories defined in the proposal. The occurrence frequency, severity, and ADR mitigation method explicitly stated in package inserts were also analyzed, as well as the top 5 specific populations with contraindications for quinolone drugs. Conclusions: Ontological representation and organization using officially approved information from drug package inserts enables the identification and bioinformatic analysis of adverse reactions caused by a specific drug with regard to predefined ADR ontology classes and semantic relations. The resulting ontology-based ADR knowledge source classifies drug-specific adverse reactions, and supports a better understanding of ADRs and safer prescription of medications. UR - https://www.jmir.org/2020/7/e20443 UR - http://dx.doi.org/10.2196/20443 UR - http://www.ncbi.nlm.nih.gov/pubmed/32706718 ID - info:doi/10.2196/20443 ER - TY - JOUR AU - Wang, Karen AU - Grossetta Nardini, Holly AU - Post, Lori AU - Edwards, Todd AU - Nunez-Smith, Marcella AU - Brandt, Cynthia PY - 2020/7/20 TI - Information Loss in Harmonizing Granular Race and Ethnicity Data: Descriptive Study of Standards JO - J Med Internet Res SP - e14591 VL - 22 IS - 7 KW - continental population groups KW - multiracial populations KW - multiethnic groups KW - data standards KW - health status disparities KW - race factors KW - demography N2 - Background: Data standards for race and ethnicity have significant implications for health equity research. Objective: We aim to describe a challenge encountered when working with a multiple?race and ethnicity assessment in the Eastern Caribbean Health Outcomes Research Network (ECHORN), a research collaborative of Barbados, Puerto Rico, Trinidad and Tobago, and the US Virgin Islands. Methods: We examined the data standards guiding harmonization of race and ethnicity data for multiracial and multiethnic populations, using the Office of Management and Budget (OMB) Statistical Policy Directive No. 15. Results: Of 1211 participants in the ECHORN cohort study, 901 (74.40%) selected 1 racial category. Of those that selected 1 category, 13.0% (117/901) selected Caribbean; 6.4% (58/901), Puerto Rican or Boricua; and 13.5% (122/901), the mixed or multiracial category. A total of 17.84% (216/1211) of participants selected 2 or more categories, with 15.19% (184/1211) selecting 2 categories and 2.64% (32/1211) selecting 3 or more categories. With aggregation of ECHORN data into OMB categories, 27.91% (338/1211) of the participants can be placed in the ?more than one race? category. Conclusions: This analysis exposes the fundamental informatics challenges that current race and ethnicity data standards present to meaningful collection, organization, and dissemination of granular data about subgroup populations in diverse and marginalized communities. Current standards should reflect the science of measuring race and ethnicity and the need for multidisciplinary teams to improve evolving standards throughout the data life cycle. UR - http://www.jmir.org/2020/7/e14591/ UR - http://dx.doi.org/10.2196/14591 UR - http://www.ncbi.nlm.nih.gov/pubmed/32706693 ID - info:doi/10.2196/14591 ER - TY - JOUR AU - Le, Nhat AU - Wiley, Matthew AU - Loza, Antonio AU - Hristidis, Vagelis AU - El-Kareh, Robert PY - 2020/7/17 TI - Prediction of Medical Concepts in Electronic Health Records: Similar Patient Analysis JO - JMIR Med Inform SP - e16008 VL - 8 IS - 7 KW - consumer health information KW - decision support techniques KW - electronic health record N2 - Background: Medicine 2.0?the adoption of Web 2.0 technologies such as social networks in health care?creates the need for apps that can find other patients with similar experiences and health conditions based on a patient?s electronic health record (EHR). Concurrently, there is an increasing number of longitudinal EHR data sets with rich information, which are essential to fulfill this need. Objective: This study aimed to evaluate the hypothesis that we can leverage similar EHRs to predict possible future medical concepts (eg, disorders) from a patient?s EHR. Methods: We represented patients? EHRs using time-based prefixes and suffixes, where each prefix or suffix is a set of medical concepts from a medical ontology. We compared the prefixes of other patients in the collection with the state of the current patient using various interpatient distance measures. The set of similar prefixes yields a set of suffixes, which we used to determine probable future concepts for the current patient?s EHR. Results: We evaluated our methods on the Multiparameter Intelligent Monitoring in Intensive Care II data set of patients, where we achieved precision up to 56.1% and recall up to 69.5%. For a limited set of clinically interesting concepts, specifically a set of procedures, we found that 86.9% (353/406) of the true-positives are clinically useful, that is, these procedures were actually performed later on the patient, and only 4.7% (19/406) of true-positives were completely irrelevant. Conclusions: These initial results indicate that predicting patients? future medical concepts is feasible. Effectively predicting medical concepts can have several applications, such as managing resources in a hospital. UR - https://medinform.jmir.org/2020/7/e16008 UR - http://dx.doi.org/10.2196/16008 UR - http://www.ncbi.nlm.nih.gov/pubmed/32706678 ID - info:doi/10.2196/16008 ER - TY - JOUR AU - Pellison, Carvalho Felipe AU - Rijo, Lopes Rui Pedro Charters AU - Lima, Costa Vinicius AU - Crepaldi, Yukie Nathalia AU - Bernardi, Andrade Filipe AU - Galliez, Mello Rafael AU - Kritski, Afrânio AU - Abhishek, Kumar AU - Alves, Domingos PY - 2020/7/6 TI - Data Integration in the Brazilian Public Health System for Tuberculosis: Use of the Semantic Web to Establish Interoperability JO - JMIR Med Inform SP - e17176 VL - 8 IS - 7 KW - health information systems KW - tuberculosis KW - ontology KW - interoperability KW - electronic health records KW - semantic web N2 - Background: Interoperability of health information systems is a challenge due to the heterogeneity of existing systems at both the technological and semantic levels of their data. The lack of existing data about interoperability disrupts intra-unit and inter-unit medical operations as well as creates challenges in conducting studies on existing data. The goal is to exchange data while providing the same meaning for data from different sources. Objective: To find ways to solve this challenge, this research paper proposes an interoperability solution for the tuberculosis treatment and follow-up scenario in Brazil using Semantic Web technology supported by an ontology. Methods: The entities of the ontology were allocated under the definitions of Basic Formal Ontology. Brazilian tuberculosis applications were tagged with entities from the resulting ontology. Results: An interoperability layer was developed to retrieve data with the same meaning and in a structured way enabling semantic and functional interoperability. Conclusions: Health professionals could use the data gathered from several data sources to enhance the effectiveness of their actions and decisions, as shown in a practical use case to integrate tuberculosis data in the State of São Paulo. UR - https://medinform.jmir.org/2020/7/e17176 UR - http://dx.doi.org/10.2196/17176 UR - http://www.ncbi.nlm.nih.gov/pubmed/32628611 ID - info:doi/10.2196/17176 ER - TY - JOUR AU - Li, Genghao AU - Li, Bing AU - Huang, Langlin AU - Hou, Sibing PY - 2020/6/23 TI - Automatic Construction of a Depression-Domain Lexicon Based on Microblogs: Text Mining Study JO - JMIR Med Inform SP - e17650 VL - 8 IS - 6 KW - depression detection KW - depression diagnosis KW - social media KW - automatic construction KW - domain-specific lexicon KW - depression lexicon KW - label propagation N2 - Background: According to a World Health Organization report in 2017, there was almost one patient with depression among every 20 people in China. However, the diagnosis of depression is usually difficult in terms of clinical detection owing to slow observation, high cost, and patient resistance. Meanwhile, with the rapid emergence of social networking sites, people tend to share their daily life and disclose inner feelings online frequently, making it possible to effectively identify mental conditions using the rich text information. There are many achievements regarding an English web-based corpus, but for research in China so far, the extraction of language features from web-related depression signals is still in a relatively primary stage. Objective: The purpose of this study was to propose an effective approach for constructing a depression-domain lexicon. This lexicon will contain language features that could help identify social media users who potentially have depression. Our study also compared the performance of detection with and without our lexicon. Methods: We autoconstructed a depression-domain lexicon using Word2Vec, a semantic relationship graph, and the label propagation algorithm. These two methods combined performed well in a specific corpus during construction. The lexicon was obtained based on 111,052 Weibo microblogs from 1868 users who were depressed or nondepressed. During depression detection, we considered six features, and we used five classification methods to test the detection performance. Results: The experiment results showed that in terms of the F1 value, our autoconstruction method performed 1% to 6% better than baseline approaches and was more effective and steadier. When applied to detection models like logistic regression and support vector machine, our lexicon helped the models outperform by 2% to 9% and was able to improve the final accuracy of potential depression detection. Conclusions: Our depression-domain lexicon was proven to be a meaningful input for classification algorithms, providing linguistic insights on the depressive status of test subjects. We believe that this lexicon will enhance early depression detection in people on social media. Future work will need to be carried out on a larger corpus and with more complex methods. UR - http://medinform.jmir.org/2020/6/e17650/ UR - http://dx.doi.org/10.2196/17650 UR - http://www.ncbi.nlm.nih.gov/pubmed/32574151 ID - info:doi/10.2196/17650 ER - TY - JOUR AU - Liu, Ziqing AU - He, Haiyang AU - Yan, Shixing AU - Wang, Yong AU - Yang, Tao AU - Li, Guo-Zheng PY - 2020/6/16 TI - End-to-End Models to Imitate Traditional Chinese Medicine Syndrome Differentiation in Lung Cancer Diagnosis: Model Development and Validation JO - JMIR Med Inform SP - e17821 VL - 8 IS - 6 KW - traditional Chinese medicine KW - syndrome differentiation KW - lung cancer KW - medical record KW - deep learning KW - model fusion N2 - Background: Traditional Chinese medicine (TCM) has been shown to be an efficient mode to manage advanced lung cancer, and accurate syndrome differentiation is crucial to treatment. Documented evidence of TCM treatment cases and the progress of artificial intelligence technology are enabling the development of intelligent TCM syndrome differentiation models. This is expected to expand the benefits of TCM to lung cancer patients. Objective: The objective of this work was to establish end-to-end TCM diagnostic models to imitate lung cancer syndrome differentiation. The proposed models used unstructured medical records as inputs to capitalize on data collected for practical TCM treatment cases by lung cancer experts. The resulting models were expected to be more efficient than approaches that leverage structured TCM datasets. Methods: We approached lung cancer TCM syndrome differentiation as a multilabel text classification problem. First, entity representation was conducted with Bidirectional Encoder Representations from Transformers and conditional random fields models. Then, five deep learning?based text classification models were applied to the construction of a medical record multilabel classifier, during which two data augmentation strategies were adopted to address overfitting issues. Finally, a fusion model approach was used to elevate the performance of the models. Results: The F1 score of the recurrent convolutional neural network (RCNN) model with augmentation was 0.8650, a 2.41% improvement over the unaugmented model. The Hamming loss for RCNN with augmentation was 0.0987, which is 1.8% lower than that of the same model without augmentation. Among the models, the text-hierarchical attention network (Text-HAN) model achieved the highest F1 scores of 0.8676 and 0.8751. The mean average precision for the word encoding?based RCNN was 10% higher than that of the character encoding?based representation. A fusion model of the text-convolutional neural network, text-recurrent neural network, and Text-HAN models achieved an F1 score of 0.8884, which showed the best performance among the models. Conclusions: Medical records could be used more productively by constructing end-to-end models to facilitate TCM diagnosis. With the aid of entity-level representation, data augmentation, and model fusion, deep learning?based multilabel classification approaches can better imitate TCM syndrome differentiation in complex cases such as advanced lung cancer. UR - https://medinform.jmir.org/2020/6/e17821 UR - http://dx.doi.org/10.2196/17821 UR - http://www.ncbi.nlm.nih.gov/pubmed/32543445 ID - info:doi/10.2196/17821 ER - TY - JOUR AU - Yu, Yue AU - Ruddy, Kathryn AU - Mansfield, Aaron AU - Zong, Nansu AU - Wen, Andrew AU - Tsuji, Shintaro AU - Huang, Ming AU - Liu, Hongfang AU - Shah, Nilay AU - Jiang, Guoqian PY - 2020/6/12 TI - Detecting and Filtering Immune-Related Adverse Events Signal Based on Text Mining and Observational Health Data Sciences and Informatics Common Data Model: Framework Development Study JO - JMIR Med Inform SP - e17353 VL - 8 IS - 6 KW - immunotherapy/adverse effects KW - drug-related side effects and adverse reactions KW - pharmacovigilance KW - adverse drug reaction reporting systems/standards KW - text mining N2 - Background: Immune checkpoint inhibitors are associated with unique immune-related adverse events (irAEs). As most of the immune checkpoint inhibitors are new to the market, it is important to conduct studies using real-world data sources to investigate their safety profiles. Objective: The aim of the study was to develop a framework for signal detection and filtration of novel irAEs for 6 Food and Drug Administration?approved immune checkpoint inhibitors. Methods: In our framework, we first used the Food and Drug Administration?s Adverse Event Reporting System (FAERS) standardized in an Observational Health Data Sciences and Informatics (OHDSI) common data model (CDM) to collect immune checkpoint inhibitor-related event data and conducted irAE signal detection. OHDSI CDM is a standard-driven data model that focuses on transforming different databases into a common format and standardizing medical terms to a common representation. We then filtered those already known irAEs from drug labels and literature by using a customized text-mining pipeline based on clinical text analysis and knowledge extraction system with Medical Dictionary for Regulatory Activities (MedDRA) as a dictionary. Finally, we classified the irAE detection results into three different categories to discover potentially new irAE signals. Results: By our text-mining pipeline, 490 irAE terms were identified from drug labels, and 918 terms were identified from the literature. In addition, of the 94 positive signals detected using CDM-based FAERS, 53 signals (56%) were labeled signals, 10 (11%) were unlabeled published signals, and 31 (33%) were potentially new signals. Conclusions: We demonstrated that our approach is effective for irAE signal detection and filtration. Moreover, our CDM-based framework could facilitate adverse drug events detection and filtration toward the goal of next-generation pharmacovigilance that seamlessly integrates electronic health record data for improved signal detection. UR - http://medinform.jmir.org/2020/6/e17353/ UR - http://dx.doi.org/10.2196/17353 UR - http://www.ncbi.nlm.nih.gov/pubmed/32530430 ID - info:doi/10.2196/17353 ER - TY - JOUR AU - Torres, Gomes Fernanda Broering AU - Gomes, Carvalho Denilsen AU - Hino, Ferreira Adriano Akira AU - Moro, Claudia AU - Cubas, Regina Marcia PY - 2020/6/9 TI - Comparison of the Results of Manual and Automated Processes of Cross-Mapping Between Nursing Terms: Quantitative Study JO - JMIR Nursing SP - e18501 VL - 3 IS - 1 KW - health information interoperability KW - nursing informatics KW - controlled vocabulary KW - standardized nursing terminology KW - ehealth N2 - Background: Cross-mapping establishes equivalence between terms from different terminology systems, which is useful for interoperability, updated terminological versions, and reuse of terms. Due to the number of terms to be mapped, this work can be extensive, tedious, and thorough, and it is susceptible to errors; this can be minimized by automated processes, which use computational tools. Objective: The aim of this study was to compare the results of manual and automated term mapping processes. Methods: In this descriptive, quantitative study, we used the results of two mapping processes as an empirical basis: manual, which used 2638 terms of nurses? records from a university hospital in southern Brazil and the International Classification for Nursing Practice (ICNP); and automated, which used the same university hospital terms and the primitive terms of the ICNP through MappICNP, an algorithm based on rules of natural language processing. The two processes were compared via equality and exclusivity assessments of new terms of the automated process and of candidate terms. Results: The automated process mapped 569/2638 (21.56%) of the source bank?s terms as identical, and the manual process mapped 650/2638 (24.63%) as identical. Regarding new terms, the automated process mapped 1031/2638 (39.08%) of the source bank?s terms as new, while the manual process mapped 1251 (47.42%). In particular, manual mapping identified 101/2638 (3.82%) terms as identical and 429 (16.26%) as new, whereas the automated process identified 20 (0.75%) terms as identical and 209 (7.92%) as new. Of the 209 terms mapped as new by the automated process, it was possible to establish an equivalence with ICNP terms in 48 (23.0%) cases. An analysis of the candidate terms offered by the automated process to the 429 new terms mapped exclusively by the manual process resulted in 100 (23.3%) candidates that had a semantic relationship with the source term. Conclusions: The automated and manual processes map identical and new terms in similar ways and can be considered complementary. Direct identification of identical terms and the offering of candidate terms through the automated process facilitate and enhance the results of the mapping; confirmation of the precision of the automated mapping requires further analysis by researchers. UR - https://nursing.jmir.org/2020/1/e18501/ UR - http://dx.doi.org/10.2196/18501 UR - http://www.ncbi.nlm.nih.gov/pubmed/34345784 ID - info:doi/10.2196/18501 ER - TY - JOUR AU - Hane, A. Christopher AU - Nori, S. Vijay AU - Crown, H. William AU - Sanghavi, M. Darshak AU - Bleicher, Paul PY - 2020/6/3 TI - Predicting Onset of Dementia Using Clinical Notes and Machine Learning: Case-Control Study JO - JMIR Med Inform SP - e17819 VL - 8 IS - 6 KW - Alzheimer disease KW - dementia KW - health information systems KW - machine learning KW - natural language processing KW - health information interoperability N2 - Background: Clinical trials need efficient tools to assist in recruiting patients at risk of Alzheimer disease and related dementias (ADRD). Early detection can also assist patients with financial planning for long-term care. Clinical notes are an important, underutilized source of information in machine learning models because of the cost of collection and complexity of analysis. Objective: This study aimed to investigate the use of deidentified clinical notes from multiple hospital systems collected over 10 years to augment retrospective machine learning models of the risk of developing ADRD. Methods: We used 2 years of data to predict the future outcome of ADRD onset. Clinical notes are provided in a deidentified format with specific terms and sentiments. Terms in clinical notes are embedded into a 100-dimensional vector space to identify clusters of related terms and abbreviations that differ across hospital systems and individual clinicians. Results: When using clinical notes, the area under the curve (AUC) improved from 0.85 to 0.94, and positive predictive value (PPV) increased from 45.07% (25,245/56,018) to 68.32% (14,153/20,717) in the model at disease onset. Models with clinical notes improved in both AUC and PPV in years 3-6 when notes? volume was largest; results are mixed in years 7 and 8 with the smallest cohorts. Conclusions: Although clinical notes helped in the short term, the presence of ADRD symptomatic terms years earlier than onset adds evidence to other studies that clinicians undercode diagnoses of ADRD. De-identified clinical notes increase the accuracy of risk models. Clinical notes collected across multiple hospital systems via natural language processing can be merged using postprocessing techniques to aid model accuracy. UR - https://medinform.jmir.org/2020/6/e17819 UR - http://dx.doi.org/10.2196/17819 UR - http://www.ncbi.nlm.nih.gov/pubmed/32490841 ID - info:doi/10.2196/17819 ER - TY - JOUR AU - Horne, Elsie AU - Tibble, Holly AU - Sheikh, Aziz AU - Tsanas, Athanasios PY - 2020/5/28 TI - Challenges of Clustering Multimodal Clinical Data: Review of Applications in Asthma Subtyping JO - JMIR Med Inform SP - e16452 VL - 8 IS - 5 KW - asthma KW - cluster analysis KW - data mining KW - machine learning KW - unsupervised machine learning N2 - Background: In the current era of personalized medicine, there is increasing interest in understanding the heterogeneity in disease populations. Cluster analysis is a method commonly used to identify subtypes in heterogeneous disease populations. The clinical data used in such applications are typically multimodal, which can make the application of traditional cluster analysis methods challenging. Objective: This study aimed to review the research literature on the application of clustering multimodal clinical data to identify asthma subtypes. We assessed common problems and shortcomings in the application of cluster analysis methods in determining asthma subtypes, such that they can be brought to the attention of the research community and avoided in future studies. Methods: We searched PubMed and Scopus bibliographic databases with terms related to cluster analysis and asthma to identify studies that applied dissimilarity-based cluster analysis methods. We recorded the analytic methods used in each study at each step of the cluster analysis process. Results: Our literature search identified 63 studies that applied cluster analysis to multimodal clinical data to identify asthma subtypes. The features fed into the cluster algorithms were of a mixed type in 47 (75%) studies and continuous in 12 (19%), and the feature type was unclear in the remaining 4 (6%) studies. A total of 23 (37%) studies used hierarchical clustering with Ward linkage, and 22 (35%) studies used k-means clustering. Of these 45 studies, 39 had mixed-type features, but only 5 specified dissimilarity measures that could handle mixed-type features. A further 9 (14%) studies used a preclustering step to create small clusters to feed on a hierarchical method. The original sample sizes in these 9 studies ranged from 84 to 349. The remaining studies used hierarchical clustering with other linkages (n=3), medoid-based methods (n=3), spectral clustering (n=1), and multiple kernel k-means clustering (n=1), and in 1 study, the methods were unclear. Of 63 studies, 54 (86%) explained the methods used to determine the number of clusters, 24 (38%) studies tested the quality of their cluster solution, and 11 (17%) studies tested the stability of their solution. Reporting of the cluster analysis was generally poor in terms of the methods employed and their justification. Conclusions: This review highlights common issues in the application of cluster analysis to multimodal clinical data to identify asthma subtypes. Some of these issues were related to the multimodal nature of the data, but many were more general issues in the application of cluster analysis. Although cluster analysis may be a useful tool for investigating disease subtypes, we recommend that future studies carefully consider the implications of clustering multimodal data, the cluster analysis process itself, and the reporting of methods to facilitate replication and interpretation of findings. UR - http://medinform.jmir.org/2020/5/e16452/ UR - http://dx.doi.org/10.2196/16452 UR - http://www.ncbi.nlm.nih.gov/pubmed/32463370 ID - info:doi/10.2196/16452 ER - TY - JOUR AU - Yu, Biyang AU - He, Zhe AU - Xing, Aiwen AU - Lustria, A. Mia Liza PY - 2020/5/21 TI - An Informatics Framework to Assess Consumer Health Language Complexity Differences: Proof-of-Concept Study JO - J Med Internet Res SP - e16795 VL - 22 IS - 5 KW - consumer health informatics KW - readability KW - digital divide KW - health literacy N2 - Background: The language gap between health consumers and health professionals has been long recognized as the main hindrance to effective health information comprehension. Although providing health information access in consumer health language (CHL) is widely accepted as the solution to the problem, health consumers are found to have varying health language preferences and proficiencies. To simplify health documents for heterogeneous consumer groups, it is important to quantify how CHLs are different in terms of complexity among various consumer groups. Objective: This study aimed to propose an informatics framework (consumer health language complexity [CHELC]) to assess the complexity differences of CHL using syntax-level, text-level, term-level, and semantic-level complexity metrics. Specifically, we identified 8 language complexity metrics validated in previous literature and combined them into a 4-faceted framework. Through a rank-based algorithm, we developed unifying scores (CHELC scores [CHELCS]) to quantify syntax-level, text-level, term-level, semantic-level, and overall CHL complexity. We applied CHELCS to compare posts of each individual on online health forums designed for (1) the general public, (2) deaf and hearing-impaired people, and (3) people with autism spectrum disorder (ASD). Methods: We examined posts with more than 4 sentences of each user from 3 health forums to understand CHL complexity differences among these groups: 12,560 posts from 3756 users in Yahoo! Answers, 25,545 posts from 1623 users in AllDeaf, and 26,484 posts from 2751 users in Wrong Planet. We calculated CHELCS for each user and compared the scores of 3 user groups (ie, deaf and hearing-impaired people, people with ASD, and the public) through 2-sample Kolmogorov-Smirnov tests and analysis of covariance tests. Results: The results suggest that users in the public forum used more complex CHL, particularly more diverse semantics and more complex health terms compared with users in the ASD and deaf and hearing-impaired user forums. However, between the latter 2 groups, people with ASD used more complex words, and deaf and hearing-impaired users used more complex syntax. Conclusions: Our results show that the users in 3 online forums had significantly different CHL complexities in different facets. The proposed framework and detailed measurements help to quantify these CHL complexity differences comprehensively. The results emphasize the importance of tailoring health-related content for different consumer groups with varying CHL complexities. UR - https://www.jmir.org/2020/5/e16795 UR - http://dx.doi.org/10.2196/16795 UR - http://www.ncbi.nlm.nih.gov/pubmed/32436849 ID - info:doi/10.2196/16795 ER - TY - JOUR AU - Falissard, Louis AU - Morgand, Claire AU - Roussel, Sylvie AU - Imbaud, Claire AU - Ghosn, Walid AU - Bounebache, Karim AU - Rey, Grégoire PY - 2020/4/28 TI - A Deep Artificial Neural Network?Based Model for Prediction of Underlying Cause of Death From Death Certificates: Algorithm Development and Validation JO - JMIR Med Inform SP - e17125 VL - 8 IS - 4 KW - machine learning KW - deep learning KW - mortality statistics KW - underlying cause of death N2 - Background: Coding of underlying causes of death from death certificates is a process that is nowadays undertaken mostly by humans with potential assistance from expert systems, such as the Iris software. It is, consequently, an expensive process that can, in addition, suffer from geospatial discrepancies, thus severely impairing the comparability of death statistics at the international level. The recent advances in artificial intelligence, specifically the rise of deep learning methods, has enabled computers to make efficient decisions on a number of complex problems that were typically considered out of reach without human assistance; they require a considerable amount of data to learn from, which is typically their main limiting factor. However, the CépiDc (Centre d?épidémiologie sur les causes médicales de Décès) stores an exhaustive database of death certificates at the French national scale, amounting to several millions of training examples available for the machine learning practitioner. Objective: This article investigates the application of deep neural network methods to coding underlying causes of death. Methods: The investigated dataset was based on data contained from every French death certificate from 2000 to 2015, containing information such as the subject?s age and gender, as well as the chain of events leading to his or her death, for a total of around 8 million observations. The task of automatically coding the subject?s underlying cause of death was then formulated as a predictive modelling problem. A deep neural network?based model was then designed and fit to the dataset. Its error rate was then assessed on an exterior test dataset and compared to the current state-of-the-art (ie, the Iris software). Statistical significance of the proposed approach?s superiority was assessed via bootstrap. Results: The proposed approach resulted in a test accuracy of 97.8% (95% CI 97.7-97.9), which constitutes a significant improvement over the current state-of-the-art and its accuracy of 74.5% (95% CI 74.0-75.0) assessed on the same test example. Such an improvement opens up a whole field of new applications, from nosologist-level batch-automated coding to international and temporal harmonization of cause of death statistics. A typical example of such an application is demonstrated by recoding French overdose-related deaths from 2000 to 2010. Conclusions: This article shows that deep artificial neural networks are perfectly suited to the analysis of electronic health records and can learn a complex set of medical rules directly from voluminous datasets, without any explicit prior knowledge. Although not entirely free from mistakes, the derived algorithm constitutes a powerful decision-making tool that is able to handle structured medical data with an unprecedented performance. We strongly believe that the methods developed in this article are highly reusable in a variety of settings related to epidemiology, biostatistics, and the medical sciences in general. UR - http://medinform.jmir.org/2020/4/e17125/ UR - http://dx.doi.org/10.2196/17125 UR - http://www.ncbi.nlm.nih.gov/pubmed/32343252 ID - info:doi/10.2196/17125 ER - TY - JOUR AU - Wang, Zheyu AU - Huang, Haoce AU - Cui, Liping AU - Chen, Juan AU - An, Jiye AU - Duan, Huilong AU - Ge, Huiqing AU - Deng, Ning PY - 2020/4/23 TI - Using Natural Language Processing Techniques to Provide Personalized Educational Materials for Chronic Disease Patients in China: Development and Assessment of a Knowledge-Based Health Recommender System JO - JMIR Med Inform SP - e17642 VL - 8 IS - 4 KW - health education KW - ontology KW - natural language processing KW - chronic disease KW - recommender system N2 - Background: Health education emerged as an important intervention for improving the awareness and self-management abilities of chronic disease patients. The development of information technologies has changed the form of patient educational materials from traditional paper materials to electronic materials. To date, the amount of patient educational materials on the internet is tremendous, with variable quality, which makes it hard to identify the most valuable materials by individuals lacking medical backgrounds. Objective: The aim of this study was to develop a health recommender system to provide appropriate educational materials for chronic disease patients in China and evaluate the effect of this system. Methods: A knowledge-based recommender system was implemented using ontology and several natural language processing (NLP) techniques. The development process was divided into 3 stages. In stage 1, an ontology was constructed to describe patient characteristics contained in the data. In stage 2, an algorithm was designed and implemented to generate recommendations based on the ontology. Patient data and educational materials were mapped to the ontology and converted into vectors of the same length, and then recommendations were generated according to similarity between these vectors. In stage 3, the ontology and algorithm were incorporated into an mHealth system for practical use. Keyword extraction algorithms and pretrained word embeddings were used to preprocess educational materials. Three strategies were proposed to improve the performance of keyword extraction. System evaluation was based on a manually assembled test collection for 50 patients and 100 educational documents. Recommendation performance was assessed using the macro precision of top-ranked documents and the overall mean average precision (MAP). Results: The constructed ontology contained 40 classes, 31 object properties, 67 data properties, and 32 individuals. A total of 80 SWRL rules were defined to implement the semantic logic of mapping patient original data to the ontology vector space. The recommender system was implemented as a separate Web service connected with patients' smartphones. According to the evaluation results, our system can achieve a macro precision up to 0.970 for the top 1 recommendation and an overall MAP score up to 0.628. Conclusions: This study demonstrated that a knowledge-based health recommender system has the potential to accurately recommend educational materials to chronic disease patients. Traditional NLP techniques combined with improvement strategies for specific language and domain proved to be effective for improving system performance. One direction for future work is to explore the effect of such systems from the perspective of patients in a practical setting. UR - http://medinform.jmir.org/2020/4/e17642/ UR - http://dx.doi.org/10.2196/17642 UR - http://www.ncbi.nlm.nih.gov/pubmed/32324148 ID - info:doi/10.2196/17642 ER - TY - JOUR AU - Lelong, Romain AU - Soualmia, F. Lina AU - Grosjean, Julien AU - Taalba, Mehdi AU - Darmoni, J. Stéfan PY - 2019/12/20 TI - Building a Semantic Health Data Warehouse in the Context of Clinical Trials: Development and Usability Study JO - JMIR Med Inform SP - e13917 VL - 7 IS - 4 KW - data warehousing KW - search engine KW - semantics KW - clinical trial KW - patient selection N2 - Background: The huge amount of clinical, administrative, and demographic data recorded and maintained by hospitals can be consistently aggregated into health data warehouses with a uniform data model. In 2017, Rouen University Hospital (RUH) initiated the design of a semantic health data warehouse enabling both semantic description and retrieval of health information. Objective: This study aimed to present a proof of concept of this semantic health data warehouse, based on the data of 250,000 patients from RUH, and to assess its ability to assist health professionals in prescreening eligible patients in a clinical trials context. Methods: The semantic health data warehouse relies on 3 distinct semantic layers: (1) a terminology and ontology portal, (2) a semantic annotator, and (3) a semantic search engine and NoSQL (not only structured query language) layer to enhance data access performances. The system adopts an entity-centered vision that provides generic search capabilities able to express data requirements in terms of the whole set of interconnected conceptual entities that compose health information. Results: We assessed the ability of the system to assist the search for 95 inclusion and exclusion criteria originating from 5 randomly chosen clinical trials from RUH. The system succeeded in fully automating 39% (29/74) of the criteria and was efficiently used as a prescreening tool for 73% (54/74) of them. Furthermore, the targeted sources of information and the search engine?related or data-related limitations that could explain the results for each criterion were also observed. Conclusions: The entity-centered vision contrasts with the usual patient-centered vision adopted by existing systems. It enables more genericity in the information retrieval process. It also allows to fully exploit the semantic description of health information. Despite their semantic annotation, searching within clinical narratives remained the major challenge of the system. A finer annotation of the clinical texts and the addition of specific functionalities would significantly improve the results. The semantic aspect of the system combined with its generic entity-centered vision enables the processing of a large range of clinical questions. However, an important part of health information remains in clinical narratives, and we are currently investigating novel approaches (deep learning) to enhance the semantic annotation of those unstructured data. UR - http://medinform.jmir.org/2019/4/e13917/ UR - http://dx.doi.org/10.2196/13917 UR - http://www.ncbi.nlm.nih.gov/pubmed/31859675 ID - info:doi/10.2196/13917 ER - TY - JOUR AU - Wu, Patrick AU - Gifford, Aliya AU - Meng, Xiangrui AU - Li, Xue AU - Campbell, Harry AU - Varley, Tim AU - Zhao, Juan AU - Carroll, Robert AU - Bastarache, Lisa AU - Denny, C. Joshua AU - Theodoratou, Evropi AU - Wei, Wei-Qi PY - 2019/11/29 TI - Mapping ICD-10 and ICD-10-CM Codes to Phecodes: Workflow Development and Initial Evaluation JO - JMIR Med Inform SP - e14325 VL - 7 IS - 4 KW - electronic health record KW - genome-wide association study KW - phenome-wide association study KW - phenotyping KW - medical informatics applications KW - data science N2 - Background: The phecode system was built upon the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) for phenome-wide association studies (PheWAS) using the electronic health record (EHR). Objective: The goal of this paper was to develop and perform an initial evaluation of maps from the International Classification of Diseases, 10th Revision (ICD-10) and the International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) codes to phecodes. Methods: We mapped ICD-10 and ICD-10-CM codes to phecodes using a number of methods and resources, such as concept relationships and explicit mappings from the Centers for Medicare & Medicaid Services, the Unified Medical Language System, Observational Health Data Sciences and Informatics, Systematized Nomenclature of Medicine-Clinical Terms, and the National Library of Medicine. We assessed the coverage of the maps in two databases: Vanderbilt University Medical Center (VUMC) using ICD-10-CM and the UK Biobank (UKBB) using ICD-10. We assessed the fidelity of the ICD-10-CM map in comparison to the gold-standard ICD-9-CM phecode map by investigating phenotype reproducibility and conducting a PheWAS. Results: We mapped >75% of ICD-10 and ICD-10-CM codes to phecodes. Of the unique codes observed in the UKBB (ICD-10) and VUMC (ICD-10-CM) cohorts, >90% were mapped to phecodes. We observed 70-75% reproducibility for chronic diseases and <10% for an acute disease for phenotypes sourced from the ICD-10-CM phecode map. Using the ICD-9-CM and ICD-10-CM maps, we conducted a PheWAS with a Lipoprotein(a) genetic variant, rs10455872, which replicated two known genotype-phenotype associations with similar effect sizes: coronary atherosclerosis (ICD-9-CM: P<.001; odds ratio (OR) 1.60 [95% CI 1.43-1.80] vs ICD-10-CM: P<.001; OR 1.60 [95% CI 1.43-1.80]) and chronic ischemic heart disease (ICD-9-CM: P<.001; OR 1.56 [95% CI 1.35-1.79] vs ICD-10-CM: P<.001; OR 1.47 [95% CI 1.22-1.77]). Conclusions: This study introduces the beta versions of ICD-10 and ICD-10-CM to phecode maps that enable researchers to leverage accumulated ICD-10 and ICD-10-CM data for PheWAS in the EHR. UR - http://medinform.jmir.org/2019/4/e14325/ UR - http://dx.doi.org/10.2196/14325 UR - http://www.ncbi.nlm.nih.gov/pubmed/31553307 ID - info:doi/10.2196/14325 ER - TY - JOUR AU - Kentgen, Markus AU - Varghese, Julian AU - Samol, Alexander AU - Waltenberger, Johannes AU - Dugas, Martin PY - 2019/08/23 TI - Common Data Elements for Acute Coronary Syndrome: Analysis Based on the Unified Medical Language System JO - JMIR Med Inform SP - e14107 VL - 7 IS - 3 KW - common data elements KW - acute coronary syndrome KW - documentation KW - standardization N2 - Background: Standardization in clinical documentation can increase efficiency and can save time and resources. Objective: The objectives of this work are to compare documentation forms for acute coronary syndrome (ACS), check for standardization, and generate a list of the most common data elements using semantic form annotation with the Unified Medical Language System (UMLS). Methods: Forms from registries, studies, risk scores, quality assurance, official guidelines, and routine documentation from four hospitals in Germany were semantically annotated using UMLS. This allowed for automatic comparison of concept frequencies and the generation of a list of the most common concepts. Results: A total of 3710 forms items from 86 sources were semantically annotated using 842 unique UMLS concepts. Half of all medical concept occurrences were covered by 60 unique concepts, which suggests the existence of a core dataset of relevant concepts. Overlap percentages between forms were relatively low, hinting at inconsistent documentation structures and lack of standardization. Conclusions: This analysis shows a lack of standardized and semantically enriched documentation for patients with ACS. Efforts made by official institutions like the European Society for Cardiology have not yet been fully implemented. Utilizing a standardized and annotated core dataset of the most important data concepts could make export and automatic reuse of data easier. The generated list of common data elements is an exemplary implementation suggestion of the concepts to use in a standardized approach. UR - http://medinform.jmir.org/2019/3/e14107/ UR - http://dx.doi.org/10.2196/14107 UR - http://www.ncbi.nlm.nih.gov/pubmed/31444871 ID - info:doi/10.2196/14107 ER - TY - JOUR AU - Holz, Christian AU - Kessler, Torsten AU - Dugas, Martin AU - Varghese, Julian PY - 2019/08/12 TI - Core Data Elements in Acute Myeloid Leukemia: A Unified Medical Language System?Based Semantic Analysis and Experts? Review JO - JMIR Med Inform SP - e13554 VL - 7 IS - 3 KW - common data elements KW - UMLS KW - acute myeloid leukemia KW - medical informatics N2 - Background: For cancer domains such as acute myeloid leukemia (AML), a large set of data elements is obtained from different institutions with heterogeneous data definitions within one patient course. The lack of clinical data harmonization impedes cross-institutional electronic data exchange and future meta-analyses. Objective: This study aimed to identify and harmonize a semantic core of common data elements (CDEs) in clinical routine and research documentation, based on a systematic metadata analysis of existing documentation models. Methods: Lists of relevant data items were collected and reviewed by hematologists from two university hospitals regarding routine documentation and several case report forms of clinical trials for AML. In addition, existing registries and international recommendations were included. Data items were coded to medical concepts via the Unified Medical Language System (UMLS) by a physician and reviewed by another physician. On the basis of the coded concepts, the data sources were analyzed for concept overlaps and identification of most frequent concepts. The most frequent concepts were then implemented as data elements in the standardized format of the Operational Data Model by the Clinical Data Interchange Standards Consortium. Results: A total of 3265 medical concepts were identified, of which 1414 were unique. Among the 1414 unique medical concepts, the 50 most frequent ones cover 26.98% of all concept occurrences within the collected AML documentation. The top 100 concepts represent 39.48% of all concepts? occurrences. Implementation of CDEs is available on a European research infrastructure and can be downloaded in different formats for reuse in different electronic data capture systems. Conclusions: Information management is a complex process for research-intense disease entities as AML that is associated with a large set of lab-based diagnostics and different treatment options. Our systematic UMLS-based analysis revealed the existence of a core data set and an exemplary reusable implementation for harmonized data capture is available on an established metadata repository. UR - http://medinform.jmir.org/2019/3/e13554/ UR - http://dx.doi.org/10.2196/13554 UR - http://www.ncbi.nlm.nih.gov/pubmed/31407666 ID - info:doi/10.2196/13554 ER - TY - JOUR AU - Lin, Chin AU - Lou, Yu-Sheng AU - Tsai, Dung-Jang AU - Lee, Chia-Cheng AU - Hsu, Chia-Jung AU - Wu, Ding-Chung AU - Wang, Mei-Chuen AU - Fang, Wen-Hui PY - 2019/7/23 TI - Projection Word Embedding Model With Hybrid Sampling Training for Classifying ICD-10-CM Codes: Longitudinal Observational Study JO - JMIR Med Inform SP - e14499 VL - 7 IS - 3 KW - word embedding KW - convolutional neural network KW - artificial intelligence KW - natural language processing KW - electronic health records N2 - Background: Most current state-of-the-art models for searching the International Classification of Diseases, Tenth Revision Clinical Modification (ICD-10-CM) codes use word embedding technology to capture useful semantic properties. However, they are limited by the quality of initial word embeddings. Word embedding trained by electronic health records (EHRs) is considered the best, but the vocabulary diversity is limited by previous medical records. Thus, we require a word embedding model that maintains the vocabulary diversity of open internet databases and the medical terminology understanding of EHRs. Moreover, we need to consider the particularity of the disease classification, wherein discharge notes present only positive disease descriptions. Objective: We aimed to propose a projection word2vec model and a hybrid sampling method. In addition, we aimed to conduct a series of experiments to validate the effectiveness of these methods. Methods: We compared the projection word2vec model and traditional word2vec model using two corpora sources: English Wikipedia and PubMed journal abstracts. We used seven published datasets to measure the medical semantic understanding of the word2vec models and used these embeddings to identify the three?character-level ICD-10-CM diagnostic codes in a set of discharge notes. On the basis of embedding technology improvement, we also tried to apply the hybrid sampling method to improve accuracy. The 94,483 labeled discharge notes from the Tri-Service General Hospital of Taipei, Taiwan, from June 1, 2015, to June 30, 2017, were used. To evaluate the model performance, 24,762 discharge notes from July 1, 2017, to December 31, 2017, from the same hospital were used. Moreover, 74,324 additional discharge notes collected from seven other hospitals were tested. The F-measure, which is the major global measure of effectiveness, was adopted. Results: In medical semantic understanding, the original EHR embeddings and PubMed embeddings exhibited superior performance to the original Wikipedia embeddings. After projection training technology was applied, the projection Wikipedia embeddings exhibited an obvious improvement but did not reach the level of original EHR embeddings or PubMed embeddings. In the subsequent ICD-10-CM coding experiment, the model that used both projection PubMed and Wikipedia embeddings had the highest testing mean F-measure (0.7362 and 0.6693 in Tri-Service General Hospital and the seven other hospitals, respectively). Moreover, the hybrid sampling method was found to improve the model performance (F-measure=0.7371/0.6698). Conclusions: The word embeddings trained using EHR and PubMed could understand medical semantics better, and the proposed projection word2vec model improved the ability of medical semantics extraction in Wikipedia embeddings. Although the improvement from the projection word2vec model in the real ICD-10-CM coding task was not substantial, the models could effectively handle emerging diseases. The proposed hybrid sampling method enables the model to behave like a human expert. UR - http://medinform.jmir.org/2019/3/e14499/ UR - http://dx.doi.org/10.2196/14499 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/14499 ER - TY - JOUR AU - Sheng, Bo AU - Huang, Liang AU - Wang, Xiangbin AU - Zhuang, Jie AU - Tang, Lihua AU - Deng, Chao AU - Zhang, Yanxin PY - 2019/07/18 TI - Identification of Knee Osteoarthritis Based on Bayesian Network: Pilot Study JO - JMIR Med Inform SP - e13562 VL - 7 IS - 3 KW - osteoarthritis KW - knee KW - classification KW - health services for the aged KW - physical fitness KW - Bayesian network N2 - Background: Early identification of knee osteoarthritis (OA) can improve treatment outcomes and reduce medical costs. However, there are major limitations among existing classification or prediction models, including abstract data processing and complicated dataset attributes, which hinder their applications in clinical practice. Objective: The aim of this study was to propose a Bayesian network (BN)?based classification model to classify people with knee OA. The proposed model can be treated as a prescreening tool, which can provide decision support for health professionals. Methods: The proposed model?s structure was based on a 3-level BN structure and then retrained by the Bayesian Search (BS) learning algorithm. The model?s parameters were determined by the expectation-maximization algorithm. The used dataset included backgrounds, the target disease, and predictors. The performance of the model was evaluated based on classification accuracy, area under the curve (AUC), specificity, sensitivity, positive predictive value (PPV), and negative predictive value (NPV); it was also compared with other well-known classification models. A test was also performed to explore whether physical fitness tests could improve the performance of the proposed model. Results: A total of 249 elderly people between the ages of 60 and 80 years, living in the Kongjiang community (Shanghai), were recruited from April to September 2007. A total of 157 instances were adopted as the dataset after data preprocessing. The experimental results showed that the results of the proposed model were higher than, or equal to, the mean scores of other classification models: .754 for accuracy, .78 for AUC, .78 for specificity, and .73 for sensitivity. The proposed model provided .45 for PPV and .92 for NPV at the prevalence of 20%. The proposed model also showed a significant improvement when compared with the traditional BN model: 6.3% increase in accuracy (from .709 to .754), 4.0% increase in AUC (from .75 to .78), 6.8% increase in specificity (from .73 to .78), 5.8% increase in sensitivity (from .69 to .73), 15.4% increase in PPV (from .39 to .45), and 2.2% increase in NPV (from .90 to .92). Furthermore, the test results showed that the performance of the proposed model could be largely enhanced through physical fitness tests in 3 evaluation indices: 10.6% increase in accuracy (from .682 to .754), 16.4% increase in AUC (from .67 to .78), and 30.0% increase in specificity (from .60 to .78). Conclusions: The proposed model presents a promising method to classify people with knee OA when compared with other classification models and the traditional BN model. It could be implemented in clinical practice as a prescreening tool for knee OA, which would not only improve the quality of health care for elderly people but also reduce overall medical costs. UR - http://medinform.jmir.org/2019/3/e13562/ UR - http://dx.doi.org/10.2196/13562 UR - http://www.ncbi.nlm.nih.gov/pubmed/31322132 ID - info:doi/10.2196/13562 ER - TY - JOUR AU - Danak, U. Shivang AU - Guetterman, C. Timothy AU - Plegue, A. Melissa AU - Holmstrom, L. Heather AU - Kadri, Reema AU - Duthler, Alexander AU - Yoo, Anne AU - Buis, R. Lorraine PY - 2019/07/11 TI - Influence of Scribes on Patient-Physician Communication in Primary Care Encounters: Mixed Methods Study JO - JMIR Med Inform SP - e14797 VL - 7 IS - 3 KW - electronic health records KW - documentation KW - medical informatics N2 - Background: With the increasing adoption of electronic health record (EHR) systems, documentation-related burdens have been increasing for health care providers. Recent estimates indicate that primary care providers spend about one-half of their workdays interacting with the EHR, of which about half is focused on clerical tasks. To reduce documentation burdens associated with the EHR, health care systems and physician practices are increasingly implementing medical scribes to assist providers with real-time documentation. Scribes are typically unlicensed paraprofessionals who assist health care providers bydocumenting notes electronically under the direction of a licensed practitioner or physician in real time. Despite the promise of scribes, few studies have investigated their effect on clinical encounters, particularly with regard to patient-provider communication. Objective: The purpose of this quasi-experimental pilot study was to understand how scribes affect patient-physician communication in primary care clinical encounters. Methods: We employed a convergent mixed methods design and included a sample of three physician-scribe pairs and 34 patients. Patients? clinical encounters were randomly assigned to a scribe or nonscribe group. We conducted patient surveys focused on perceptions of patient-provider communication and satisfaction with encounters, video recorded clinical encounters, and conducted physician interviews about their experiences with scribes. Results: Overall, the survey results revealed that patients across both arms reported very high satisfaction of communication with their physician, their physician?s use of the EHR, and their care, with very little variability. Video recording analysis supported patient survey data by demonstrating high measures of communication among physicians in both scribed and nonscribed encounters. Furthermore, video recordings revealed that the presence of scribes had very little effect on the clinical encounter. Conclusions: From the patient?s perspective, scribes are an acceptable addition to clinical encounters. Although they do not have much impact on patients? perceptions of satisfaction and their impact on the clinical encounter itself was minimal, their potential to reduce documentation-related burden on physicians is valuable. Physicians noted important issues related to scribes, including important considerations for implementing scribe programs, the role of scribes in patient interactions, how physicians work with scribes, characteristics of good scribes, and the role of scribes in physician workflow. UR - http://medinform.jmir.org/2019/3/e14797/ UR - http://dx.doi.org/10.2196/14797 UR - http://www.ncbi.nlm.nih.gov/pubmed/31298218 ID - info:doi/10.2196/14797 ER - TY - JOUR AU - Block, J. Lorraine AU - Currie, M. Leanne AU - Hardiker, R. Nicholas AU - Strudwick, Gillian PY - 2019/06/26 TI - Visibility of Community Nursing Within an Administrative Health Classification System: Evaluation of Content Coverage JO - J Med Internet Res SP - e12847 VL - 21 IS - 6 KW - World Health Organization KW - classification KW - nursing informatics KW - medical informatics KW - data collection KW - terminology KW - community health services KW - standardized nursing terminology N2 - Background: The World Health Organization is in the process of developing an international administrative classification for health called the International Classification of Health Interventions (ICHI). The purpose of ICHI is to provide a tool for supporting intervention reporting and analysis at a global level for policy development and beyond. Nurses represent the largest resource carrying out clinical interventions in any health system. With the shift in nursing care from hospital to community settings in many countries, it is important to ensure that community nursing interventions are present in any international health information system. Thus, an investigation into the extent to which community nursing interventions were covered in ICHI was needed. Objective: The objectives of this study were to examine the extent to which International Classification for Nursing Practice (ICNP) community nursing interventions were represented in the ICHI administrative classification system, to identify themes related to gaps in coverage, and to support continued advancements in understanding the complexities of knowledge representation in standardized clinical terminologies and classifications. Methods: This descriptive study used a content mapping approach in 2 phases in 2018. A total of 187 nursing intervention codes were extracted from the ICNP Community Nursing Catalogue and mapped to ICHI. In phase 1, 2 coders completed independent mapping activities. In phase 2, the 2 coders compared each list and discussed concept matches until consensus on ICNP-ICHI match and on mapping relationship was reached. Results: The initial percentage agreement between the 2 coders was 47% (n=88), but reached 100% with consensus processes. After consensus was reached, 151 (81%) of the community nursing interventions resulted in an ICHI match. A total of 36 (19%) of community nursing interventions had no match to ICHI content. A total of 100 (53%) community nursing interventions resulted in a broader ICHI code, 9 (5%) resulted in a narrower ICHI code, and 42 (23%) were considered equivalent. ICNP concepts that were not represented in ICHI were thematically grouped into the categories family and caregivers, death and dying, and case management. Conclusions: Overall, the content mapping yielded similar results to other content mapping studies in nursing. However, it also found areas of missing concept coverage, difficulties with interterminology mapping, and further need to develop mapping methods. UR - https://www.jmir.org/2019/6/e12847/ UR - http://dx.doi.org/10.2196/12847 UR - http://www.ncbi.nlm.nih.gov/pubmed/31244480 ID - info:doi/10.2196/12847 ER - TY - JOUR AU - Foufi, Vasiliki AU - Timakum, Tatsawan AU - Gaudet-Blavignac, Christophe AU - Lovis, Christian AU - Song, Min PY - 2019/6/13 TI - Mining of Textual Health Information from Reddit: Analysis of Chronic Diseases With Extracted Entities and Their Relations JO - J Med Internet Res SP - e12876 VL - 21 IS - 6 KW - social media KW - chronic disease KW - data mining N2 - Background: Social media platforms constitute a rich data source for natural language processing tasks such as named entity recognition, relation extraction, and sentiment analysis. In particular, social media platforms about health provide a different insight into patient?s experiences with diseases and treatment than those found in the scientific literature. Objective: This paper aimed to report a study of entities related to chronic diseases and their relation in user-generated text posts. The major focus of our research is the study of biomedical entities found in health social media platforms and their relations and the way people suffering from chronic diseases express themselves. Methods: We collected a corpus of 17,624 text posts from disease-specific subreddits of the social news and discussion website Reddit. For entity and relation extraction from this corpus, we employed the PKDE4J tool developed by Song et al (2015). PKDE4J is a text mining system that integrates dictionary-based entity extraction and rule-based relation extraction in a highly flexible and extensible framework. Results: Using PKDE4J, we extracted 2 types of entities and relations: biomedical entities and relations and subject-predicate-object entity relations. In total, 82,138 entities and 30,341 relation pairs were extracted from the Reddit dataset. The most highly mentioned entities were those related to oncological disease (2884 occurrences of cancer) and asthma (2180 occurrences). The relation pair anatomy-disease was the most frequent (5550 occurrences), the highest frequent entities in this pair being cancer and lymph. The manual validation of the extracted entities showed a very good performance of the system at the entity extraction task (3682/5151, 71.48% extracted entities were correctly labeled). Conclusions: This study showed that people are eager to share their personal experience with chronic diseases on social media platforms despite possible privacy and security issues. The results reported in this paper are promising and demonstrate the need for more in-depth studies on the way patients with chronic diseases express themselves on social media platforms. UR - http://www.jmir.org/2019/6/e12876/ UR - http://dx.doi.org/10.2196/12876 UR - http://www.ncbi.nlm.nih.gov/pubmed/31199327 ID - info:doi/10.2196/12876 ER - TY - JOUR AU - On, Jeongah AU - Park, Hyeoun-Ae AU - Song, Tae-Min PY - 2019/6/7 TI - Sentiment Analysis of Social Media on Childhood Vaccination: Development of an Ontology JO - J Med Internet Res SP - e13456 VL - 21 IS - 6 KW - social media KW - vaccination KW - health information interoperability KW - semantics N2 - Background: Although vaccination rates are above the threshold for herd immunity in South Korea, a growing number of parents have expressed concerns about the safety of vaccines. It is important to understand these concerns so that we can maintain high vaccination rates. Objective: The aim of this study was to develop a childhood vaccination ontology to serve as a framework for collecting and analyzing social data on childhood vaccination and to use this ontology for identifying concerns about and sentiments toward childhood vaccination from social data. Methods: The domain and scope of the ontology were determined by developing competency questions. We checked if existing ontologies and conceptual frameworks related to vaccination can be reused for the childhood vaccination ontology. Terms were collected from clinical practice guidelines, research papers, and posts on social media platforms. Class concepts were extracted from these terms. A class hierarchy was developed using a top-down approach. The ontology was evaluated in terms of description logics, face and content validity, and coverage. In total, 40,359 Korean posts on childhood vaccination were collected from 27 social media channels between January and December 2015. Vaccination issues were identified and classified using the second-level class concepts of the ontology. The sentiments were classified in 3 ways: positive, negative or neutral. Posts were analyzed using frequency, trend, logistic regression, and association rules. Results: Our childhood vaccination ontology comprised 9 superclasses with 137 subclasses and 431 synonyms for class, attribute, and value concepts. Parent?s health belief appeared in 53.21% (15,709/29,521) of posts and positive sentiments appeared in 64.08% (17,454/27,236) of posts. Trends in sentiments toward vaccination were affected by news about vaccinations. Posts with parents? health belief, vaccination availability, and vaccination policy were associated with positive sentiments, whereas posts with experience of vaccine adverse events were associated with negative sentiments. Conclusions: The childhood vaccination ontology developed in this study was useful for collecting and analyzing social data on childhood vaccination. We expect that practitioners and researchers in the field of childhood vaccination could use our ontology to identify concerns about and sentiments toward childhood vaccination from social data. UR - http://www.jmir.org/2019/6/e13456/ UR - http://dx.doi.org/10.2196/13456 UR - http://www.ncbi.nlm.nih.gov/pubmed/31199290 ID - info:doi/10.2196/13456 ER - TY - JOUR PY - 2019// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e9695 VL - 11 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v11i1.9695 ID - info:doi/10.5210/ojphi.v11i1.9695 ER - TY - JOUR AU - Gu, Gen AU - Zhang, Xingting AU - Zhu, Xingeng AU - Jian, Zhe AU - Chen, Ken AU - Wen, Dong AU - Gao, Li AU - Zhang, Shaodian AU - Wang, Fei AU - Ma, Handong AU - Lei, Jianbo PY - 2019/05/23 TI - Development of a Consumer Health Vocabulary by Mining Health Forum Texts Based on Word Embedding: Semiautomatic Approach JO - JMIR Med Inform SP - e12704 VL - 7 IS - 2 KW - consumer health vocabulary KW - word embedding KW - representation learning KW - natural language processing KW - consumer health information KW - ontology enrichment N2 - Background: The vocabulary gap between consumers and professionals in the medical domain hinders information seeking and communication. Consumer health vocabularies have been developed to aid such informatics applications. This purpose is best served if the vocabulary evolves with consumers? language. Objective: Our objective is to develop a method for identifying and adding new terms to consumer health vocabularies, so that it can keep up with the constantly evolving medical knowledge and language use. Methods: In this paper, we propose a consumer health term?finding framework based on a distributed word vector space model. We first learned word vectors from a large-scale text corpus and then adopted a supervised method with existing consumer health vocabularies for learning vector representation of words, which can provide additional supervised fine tuning after unsupervised word embedding learning. With a fine-tuned word vector space, we identified pairs of professional terms and their consumer variants by their semantic distance in the vector space. A subsequent manual review of the extracted and labeled pairs of entities was conducted to validate the results generated by the proposed approach. The results were evaluated using mean reciprocal rank (MRR). Results: Manual evaluation showed that it is feasible to identify alternative medical concepts by using professional or consumer concepts as queries in the word vector space without fine tuning, but the results are more promising in the final fine-tuned word vector space. The MRR values indicated that on an average, a professional or consumer concept is about 14th closest to its counterpart in the word vector space without fine tuning, and the MRR in the final fine-tuned word vector space is 8. Furthermore, the results demonstrate that our method can collect abbreviations and common typos frequently used by consumers. Conclusions: By integrating a large amount of text information and existing consumer health vocabularies, our method outperformed several baseline ranking methods and is effective for generating a list of candidate terms for human review during consumer health vocabulary development. UR - http://medinform.jmir.org/2019/2/e12704/ UR - http://dx.doi.org/10.2196/12704 UR - http://www.ncbi.nlm.nih.gov/pubmed/31124461 ID - info:doi/10.2196/12704 ER - TY - JOUR AU - Brenas, Hael Jon AU - Shin, Kyong Eun AU - Shaban-Nejad, Arash PY - 2019/05/21 TI - Adverse Childhood Experiences Ontology for Mental Health Surveillance, Research, and Evaluation: Advanced Knowledge Representation and Semantic Web Techniques JO - JMIR Ment Health SP - e13498 VL - 6 IS - 5 KW - ontologies KW - mental health surveillance KW - adverse childhood experiences KW - semantics KW - computational psychiatry N2 - Background: Adverse Childhood Experiences (ACEs), a set of negative events and processes that a person might encounter during childhood and adolescence, have been proven to be linked to increased risks of a multitude of negative health outcomes and conditions when children reach adulthood and beyond. Objective: To better understand the relationship between ACEs and their relevant risk factors with associated health outcomes and to eventually design and implement preventive interventions, access to an integrated coherent dataset is needed. Therefore, we implemented a formal ontology as a resource to allow the mental health community to facilitate data integration and knowledge modeling and to improve ACEs? surveillance and research. Methods: We use advanced knowledge representation and semantic Web tools and techniques to implement the ontology. The current implementation of the ontology is expressed in the description logic ALCRIQ(D), a sublogic of Web Ontology Language (OWL 2). Results: The ACEs Ontology has been implemented and made available to the mental health community and the public via the BioPortal repository. Moreover, multiple use-case scenarios have been introduced to showcase and evaluate the usability of the ontology in action. The ontology was created to be used by major actors in the ACEs community with different applications, from the diagnosis of individuals and predicting potential negative outcomes that they might encounter to the prevention of ACEs in a population and designing interventions and policies. Conclusions: The ACEs Ontology provides a uniform and reusable semantic network and an integrated knowledge structure for mental health practitioners and researchers to improve ACEs? surveillance and evaluation. UR - http://mental.jmir.org/2019/5/e13498/ UR - http://dx.doi.org/10.2196/13498 UR - http://www.ncbi.nlm.nih.gov/pubmed/31115344 ID - info:doi/10.2196/13498 ER - TY - JOUR AU - Odigie, Eseosa AU - Lacson, Ronilda AU - Raja, Ali AU - Osterbur, David AU - Ip, Ivan AU - Schneider, Louise AU - Khorasani, Ramin PY - 2019/05/13 TI - Fast Healthcare Interoperability Resources, Clinical Quality Language, and Systematized Nomenclature of Medicine?Clinical Terms in Representing Clinical Evidence Logic Statements for the Use of Imaging Procedures: Descriptive Study JO - JMIR Med Inform SP - e13590 VL - 7 IS - 2 KW - knowledge representation KW - guidelines KW - evidence-based medicine KW - clinical decision support N2 - Background: Evidence-based guidelines and recommendations can be transformed into ?If-Then? Clinical Evidence Logic Statements (CELS). Imaging-related CELS were represented in standardized formats in the Harvard Medical School Library of Evidence (HLE). Objective: We aimed to (1) describe the representation of CELS using established Systematized Nomenclature of Medicine?Clinical Terms (SNOMED CT), Clinical Quality Language (CQL), and Fast Healthcare Interoperability Resources (FHIR) standards and (2) assess the limitations of using these standards to represent imaging-related CELS. Methods: This study was exempt from review by the Institutional Review Board as it involved no human subjects. Imaging-related clinical recommendations were extracted from evidence sources and translated into CELS. The clinical terminologies of CELS were represented using SNOMED CT and the condition-action logic was represented in CQL and FHIR. Numbers of fully and partially represented CELS were tallied. Results: A total of 765 CELS were represented in the HLE as of December 2018. We were able to fully represent 137 of 765 (17.9%) CELS using SNOMED CT, CQL, and FHIR. We were able to represent terms using SNOMED CT in the temporal component for action (?Then?) statements in CQL and FHIR in 755 of 765 (98.7%) CELS. Conclusions: CELS were represented as shareable clinical decision support (CDS) knowledge artifacts using existing standards?SNOMED CT, FHIR, and CQL?to promote and accelerate adoption of evidence-based practice. Limitations to standardization persist, which could be minimized with an add-on set of standard terms and value sets and by adding time frames to the CQL framework. UR - http://medinform.jmir.org/2019/2/e13590/ UR - http://dx.doi.org/10.2196/13590 UR - http://www.ncbi.nlm.nih.gov/pubmed/31094359 ID - info:doi/10.2196/13590 ER - TY - JOUR AU - Arbabi, Aryan AU - Adams, R. David AU - Fidler, Sanja AU - Brudno, Michael PY - 2019/05/10 TI - Identifying Clinical Terms in Medical Text Using Ontology-Guided Machine Learning JO - JMIR Med Inform SP - e12596 VL - 7 IS - 2 KW - concept recognition KW - medical text mining KW - biomedical ontologies KW - machine learning KW - phenotyping KW - human phenotype ontology N2 - Background: Automatic recognition of medical concepts in unstructured text is an important component of many clinical and research applications, and its accuracy has a large impact on electronic health record analysis. The mining of medical concepts is complicated by the broad use of synonyms and nonstandard terms in medical documents. Objective: We present a machine learning model for concept recognition in large unstructured text, which optimizes the use of ontological structures and can identify previously unobserved synonyms for concepts in the ontology. Methods: We present a neural dictionary model that can be used to predict if a phrase is synonymous to a concept in a reference ontology. Our model, called the Neural Concept Recognizer (NCR), uses a convolutional neural network to encode input phrases and then rank medical concepts based on the similarity in that space. It uses the hierarchical structure provided by the biomedical ontology as an implicit prior embedding to better learn embedding of various terms. We trained our model on two biomedical ontologies?the Human Phenotype Ontology (HPO) and Systematized Nomenclature of Medicine - Clinical Terms (SNOMED-CT). Results: We tested our model trained on HPO by using two different data sets: 288 annotated PubMed abstracts and 39 clinical reports. We achieved 1.7%-3% higher F1-scores than those for our strongest manually engineered rule-based baselines (P=.003). We also tested our model trained on the SNOMED-CT by using 2000 Intensive Care Unit discharge summaries from MIMIC (Multiparameter Intelligent Monitoring in Intensive Care) and achieved 0.9%-1.3% higher F1-scores than those of our baseline. The results of our experiments show high accuracy of our model as well as the value of using the taxonomy structure of the ontology in concept recognition. Conclusion: Most popular medical concept recognizers rely on rule-based models, which cannot generalize well to unseen synonyms. In addition, most machine learning methods typically require large corpora of annotated text that cover all classes of concepts, which can be extremely difficult to obtain for biomedical ontologies. Without relying on large-scale labeled training data or requiring any custom training, our model can be efficiently generalized to new synonyms and performs as well or better than state-of-the-art methods custom built for specific ontologies. UR - http://medinform.jmir.org/2019/2/e12596/ UR - http://dx.doi.org/10.2196/12596 UR - http://www.ncbi.nlm.nih.gov/pubmed/31094361 ID - info:doi/10.2196/12596 ER - TY - JOUR AU - Kim, Hyeoneui AU - Mentzer, Jessica AU - Taira, Ricky PY - 2019/04/23 TI - Developing a Physical Activity Ontology to Support the Interoperability of Physical Activity Data JO - J Med Internet Res SP - e12776 VL - 21 IS - 4 KW - exercise KW - leisure activities KW - health information interoperability KW - terminology as topic N2 - Background: Physical activity data provides important information on disease onset, progression, and treatment outcomes. Although analyzing physical activity data in conjunction with other clinical and microbiological data will lead to new insights crucial for improving human health, it has been hampered partly because of the large variations in the way the data are collected and presented. Objective: The aim of this study was to develop a Physical Activity Ontology (PACO) to support structuring and standardizing heterogeneous descriptions of physical activities. Methods: We prepared a corpus of 1140 unique sentences collected from various physical activity questionnaires and scales as well as existing standardized terminologies and ontologies. We extracted concepts relevant to physical activity from the corpus using a natural language processing toolkit called Multipurpose Text Processing Tool. The target concepts were formalized into an ontology using Protégé (version 4). Evaluation of PACO was performed to ensure logical and structural consistency as well as adherence to the best practice principles of building an ontology. A use case application of PACO was demonstrated by structuring and standardizing 36 exercise habit statements and then automatically classifying them to a defined class of either sufficiently active or insufficiently active using FaCT++, an ontology reasoner available in Protégé. Results: PACO was constructed using 268 unique concepts extracted from the questionnaires and assessment scales. PACO contains 225 classes including 9 defined classes, 20 object properties, 1 data property, and 23 instances (excluding 36 exercise statements). The maximum depth of classes is 4, and the maximum number of siblings is 38. The evaluations with ontology auditing tools confirmed that PACO is structurally and logically consistent and satisfies the majority of the best practice rules of ontology authoring. We showed in a small sample of 36 exercise habit statements that we could formally represent them using PACO concepts and object properties. The formal representation was used to infer a patient activity status category of sufficiently active or insufficiently active using the FaCT++ reasoner. Conclusions: As a first step toward standardizing and structuring heterogeneous descriptions of physical activities for integrative data analyses, PACO was constructed based on the concepts collected from physical activity questionnaires and assessment scales. PACO was evaluated to be structurally consistent and compliant to ontology authoring principles. PACO was also demonstrated to be potentially useful in standardizing heterogeneous physical activity descriptions and classifying them into clinically meaningful categories that reflect adequacy of exercise. UR - http://www.jmir.org/2019/4/e12776/ UR - http://dx.doi.org/10.2196/12776 UR - http://www.ncbi.nlm.nih.gov/pubmed/31012864 ID - info:doi/10.2196/12776 ER - TY - JOUR AU - Shin, Jeong Seo AU - You, Chan Seng AU - Park, Rang Yu AU - Roh, Jin AU - Kim, Jang-Hee AU - Haam, Seokjin AU - Reich, G. Christian AU - Blacketer, Clair AU - Son, Dae-Soon AU - Oh, Seungbin AU - Park, Woong Rae PY - 2019/03/26 TI - Genomic Common Data Model for Seamless Interoperation of Biomedical Data in Clinical Practice: Retrospective Study JO - J Med Internet Res SP - e13249 VL - 21 IS - 3 KW - high-throughput nucleotide sequencing KW - databases, genetic KW - multicenter study KW - patient privacy KW - data visualization N2 - Background: Clinical sequencing data should be shared in order to achieve the sufficient scale and diversity required to provide strong evidence for improving patient care. A distributed research network allows researchers to share this evidence rather than the patient-level data across centers, thereby avoiding privacy issues. The Observational Medical Outcomes Partnership (OMOP) common data model (CDM) used in distributed research networks has low coverage of sequencing data and does not reflect the latest trends of precision medicine. Objective: The aim of this study was to develop and evaluate the feasibility of a genomic CDM (G-CDM), as an extension of the OMOP-CDM, for application of genomic data in clinical practice. Methods: Existing genomic data models and sequencing reports were reviewed to extend the OMOP-CDM to cover genomic data. The Human Genome Organisation Gene Nomenclature Committee and Human Genome Variation Society nomenclature were adopted to standardize the terminology in the model. Sequencing data of 114 and 1060 patients with lung cancer were obtained from the Ajou University School of Medicine database of Ajou University Hospital and The Cancer Genome Atlas, respectively, which were transformed to a format appropriate for the G-CDM. The data were compared with respect to gene name, variant type, and actionable mutations. Results: The G-CDM was extended into four tables linked to tables of the OMOP-CDM. Upon comparison with The Cancer Genome Atlas data, a clinically actionable mutation, p.Leu858Arg, in the EGFR gene was 6.64 times more frequent in the Ajou University School of Medicine database, while the p.Gly12Xaa mutation in the KRAS gene was 2.02 times more frequent in The Cancer Genome Atlas dataset. The data-exploring tool GeneProfiler was further developed to conduct descriptive analyses automatically using the G-CDM, which provides the proportions of genes, variant types, and actionable mutations. GeneProfiler also allows for querying the specific gene name and Human Genome Variation Society nomenclature to calculate the proportion of patients with a given mutation. Conclusions: We developed the G-CDM for effective integration of genomic data with standardized clinical data, allowing for data sharing across institutes. The feasibility of the G-CDM was validated by assessing the differences in data characteristics between two different genomic databases through the proposed data-exploring tool GeneProfiler. The G-CDM may facilitate analyses of interoperating clinical and genomic datasets across multiple institutions, minimizing privacy issues and enabling researchers to better understand the characteristics of patients and promote personalized medicine in clinical practice. UR - http://www.jmir.org/2019/3/e13249/ UR - http://dx.doi.org/10.2196/13249 UR - http://www.ncbi.nlm.nih.gov/pubmed/30912749 ID - info:doi/10.2196/13249 ER - TY - JOUR AU - Chu, Ling AU - Kannan, Vaishnavi AU - Basit, A. Mujeeb AU - Schaeflein, J. Diane AU - Ortuzar, R. Adolfo AU - Glorioso, F. Jimmie AU - Buchanan, R. Joel AU - Willett, L. Duwayne PY - 2019/01/16 TI - SNOMED CT Concept Hierarchies for Computable Clinical Phenotypes From Electronic Health Record Data: Comparison of Intensional Versus Extensional Value Sets JO - JMIR Med Inform SP - e11487 VL - 7 IS - 1 KW - SNOMED CT KW - value sets KW - clinical phenotypes KW - population health KW - pragmatic clinical study N2 - Background: Defining clinical phenotypes from electronic health record (EHR)?derived data proves crucial for clinical decision support, population health endeavors, and translational research. EHR diagnoses now commonly draw from a finely grained clinical terminology?either native SNOMED CT or a vendor-supplied terminology mapped to SNOMED CT concepts as the standard for EHR interoperability. Accordingly, electronic clinical quality measures (eCQMs) increasingly define clinical phenotypes with SNOMED CT value sets. The work of creating and maintaining list-based value sets proves daunting, as does insuring that their contents accurately represent the clinically intended condition. Objective: The goal of the research was to compare an intensional (concept hierarchy-based) versus extensional (list-based) value set approach to defining clinical phenotypes using SNOMED CT?encoded data from EHRs by evaluating value set conciseness, time to create, and completeness. Methods: Starting from published Centers for Medicare and Medicaid Services (CMS) high-priority eCQMs, we selected 10 clinical conditions referenced by those eCQMs. For each, the published SNOMED CT list-based (extensional) value set was downloaded from the Value Set Authority Center (VSAC). Ten corresponding SNOMED CT hierarchy-based intensional value sets for the same conditions were identified within our EHR. From each hierarchy-based intensional value set, an exactly equivalent full extensional value set was derived enumerating all included descendant SNOMED CT concepts. Comparisons were then made between (1) VSAC-downloaded list-based (extensional) value sets, (2) corresponding hierarchy-based intensional value sets for the same conditions, and (3) derived list-based (extensional) value sets exactly equivalent to the hierarchy-based intensional value sets. Value set conciseness was assessed by the number of SNOMED CT concepts needed for definition. Time to construct the value sets for local use was measured. Value set completeness was assessed by comparing contents of the downloaded extensional versus intensional value sets. Two measures of content completeness were made: for individual SNOMED CT concepts and for the mapped diagnosis clinical terms available for selection within the EHR by clinicians. Results: The 10 hierarchy-based intensional value sets proved far simpler and faster to construct than exactly equivalent derived extensional value set lists, requiring a median 3 versus 78 concepts to define and 5 versus 37 minutes to build. The hierarchy-based intensional value sets also proved more complete: in comparison, the 10 downloaded 2018 extensional value sets contained a median of just 35% of the intensional value sets? SNOMED CT concepts and 65% of mapped EHR clinical terms. Conclusions: In the EHR era, defining conditions preferentially should employ SNOMED CT concept hierarchy-based (intensional) value sets rather than extensional lists. By doing so, clinical guideline and eCQM authors can more readily engage specialists in vetting condition subtypes to include and exclude, and streamline broad EHR implementation of condition-specific decision support promoting guideline adherence for patient benefit. UR - http://medinform.jmir.org/2019/1/e11487/ UR - http://dx.doi.org/10.2196/11487 UR - http://www.ncbi.nlm.nih.gov/pubmed/30664458 ID - info:doi/10.2196/11487 ER - TY - JOUR AU - Jing, Xia AU - Hardiker, R. Nicholas AU - Kay, Stephen AU - Gao, Yongsheng PY - 2018/12/21 TI - Identifying Principles for the Construction of an Ontology-Based Knowledge Base: A Case Study Approach JO - JMIR Med Inform SP - e52 VL - 6 IS - 4 KW - cystic fibrosis KW - knowledge base KW - knowledge representation KW - molecular genetics information KW - ontology KW - OntoKBCF KW - phenotypes N2 - Background: Ontologies are key enabling technologies for the Semantic Web. The Web Ontology Language (OWL) is a semantic markup language for publishing and sharing ontologies. Objective: The supply of customizable, computable, and formally represented molecular genetics information and health information, via electronic health record (EHR) interfaces, can play a critical role in achieving precision medicine. In this study, we used cystic fibrosis as an example to build an Ontology-based Knowledge Base prototype on Cystic Fibrobis (OntoKBCF) to supply such information via an EHR prototype. In addition, we elaborate on the construction and representation principles, approaches, applications, and representation challenges that we faced in the construction of OntoKBCF. The principles and approaches can be referenced and applied in constructing other ontology-based domain knowledge bases. Methods: First, we defined the scope of OntoKBCF according to possible clinical information needs about cystic fibrosis on both a molecular level and a clinical phenotype level. We then selected the knowledge sources to be represented in OntoKBCF. We utilized top-to-bottom content analysis and bottom-up construction to build OntoKBCF. Protégé-OWL was used to construct OntoKBCF. The construction principles included (1) to use existing basic terms as much as possible; (2) to use intersection and combination in representations; (3) to represent as many different types of facts as possible; and (4) to provide 2-5 examples for each type. HermiT 1.3.8.413 within Protégé-5.1.0 was used to check the consistency of OntoKBCF. Results: OntoKBCF was constructed successfully, with the inclusion of 408 classes, 35 properties, and 113 equivalent classes. OntoKBCF includes both atomic concepts (such as amino acid) and complex concepts (such as ?adolescent female cystic fibrosis patient?) and their descriptions. We demonstrated that OntoKBCF could make customizable molecular and health information available automatically and usable via an EHR prototype. The main challenges include the provision of a more comprehensive account of different patient groups as well as the representation of uncertain knowledge, ambiguous concepts, and negative statements and more complicated and detailed molecular mechanisms or pathway information about cystic fibrosis. Conclusions: Although cystic fibrosis is just one example, based on the current structure of OntoKBCF, it should be relatively straightforward to extend the prototype to cover different topics. Moreover, the principles underpinning its development could be reused for building alternative human monogenetic diseases knowledge bases. UR - http://medinform.jmir.org/2018/4/e52/ UR - http://dx.doi.org/10.2196/medinform.9979 UR - http://www.ncbi.nlm.nih.gov/pubmed/30578220 ID - info:doi/10.2196/medinform.9979 ER - TY - JOUR AU - Varghese, Julian AU - Sandmann, Sarah AU - Dugas, Martin PY - 2018/10/15 TI - Web-Based Information Infrastructure Increases the Interrater Reliability of Medical Coders: Quasi-Experimental Study JO - J Med Internet Res SP - e274 VL - 20 IS - 10 KW - clinical coding KW - health information interoperability KW - Unified Medical Language System KW - eligibility criteria N2 - Background: Medical coding is essential for standardized communication and integration of clinical data. The Unified Medical Language System by the National Library of Medicine is the largest clinical terminology system for medical coders and Natural Language Processing tools. However, the abundance of ambiguous codes leads to low rates of uniform coding among different coders. Objective: The objective of our study was to measure uniform coding among different medical experts in terms of interrater reliability and analyze the effect on interrater reliability using an expert- and Web-based code suggestion system. Methods: We conducted a quasi-experimental study in which 6 medical experts coded 602 medical items from structured quality assurance forms or free-text eligibility criteria of 20 different clinical trials. The medical item content was selected on the basis of mortality-leading diseases according to World Health Organization data. The intervention comprised using a semiautomatic code suggestion tool that is linked to a European information infrastructure providing a large medical text corpus of >300,000 medical form items with expert-assigned semantic codes. Krippendorff alpha (Kalpha) with bootstrap analysis was used for the interrater reliability analysis, and coding times were measured before and after the intervention. Results: The intervention improved interrater reliability in structured quality assurance form items (from Kalpha=0.50, 95% CI 0.43-0.57 to Kalpha=0.62 95% CI 0.55-0.69) and free-text eligibility criteria (from Kalpha=0.19, 95% CI 0.14-0.24 to Kalpha=0.43, 95% CI 0.37-0.50) while preserving or slightly reducing the mean coding time per item for all 6 coders. Regardless of the intervention, precoordination and structured items were associated with significantly high interrater reliability, but the proportion of items that were precoordinated significantly increased after intervention (eligibility criteria: OR 4.92, 95% CI 2.78-8.72; quality assurance: OR 1.96, 95% CI 1.19-3.25). Conclusions: The Web-based code suggestion mechanism improved interrater reliability toward moderate or even substantial intercoder agreement. Precoordination and the use of structured versus free-text data elements are key drivers of higher interrater reliability. UR - http://www.jmir.org/2018/10/e274/ UR - http://dx.doi.org/10.2196/jmir.9644 UR - http://www.ncbi.nlm.nih.gov/pubmed/30322834 ID - info:doi/10.2196/jmir.9644 ER - TY - JOUR AU - Brajovic, Sonja AU - Blaser, A. David AU - Zisk, Meaghan AU - Caligtan, Christine AU - Okun, Sally AU - Hall, Marni AU - Pamer, A. Carol PY - 2018/08/21 TI - Validating a Framework for Coding Patient-Reported Health Information to the Medical Dictionary for Regulatory Activities Terminology: An Evaluative Study JO - JMIR Med Inform SP - e42 VL - 6 IS - 3 KW - adverse drug events KW - Food and Drug Administration KW - MedDRA KW - patient-generated health data KW - PatientsLikeMe KW - vocabulary, controlled KW - data curation N2 - Background: The availability of and interest in patient-generated health data (PGHD) have grown steadily. Patients describe medical experiences differently compared with how clinicians or researchers would describe their observations of those same experiences. Patients may find nonserious, known adverse drug events (ADEs) to be an ongoing concern, which impacts the tolerability and adherence. Clinicians must be vigilant for medically serious, potentially fatal ADEs. Having both perspectives provides patients and clinicians with a complete picture of what to expect from drug therapies. Multiple initiatives seek to incorporate patients? perspectives into drug development, including PGHD exploration for pharmacovigilance. The Food and Drug Administration (FDA) Adverse Event Reporting System contains case reports of postmarketing ADEs. To facilitate the analysis of these case reports, case details are coded using the Medical Dictionary for Regulatory Activities (MedDRA). PatientsLikeMe is a Web-based network where patients report, track, share, and discuss their health information. PatientsLikeMe captures PGHD through free-text and structured data fields. PatientsLikeMe structured data are coded to multiple medical terminologies, including MedDRA. The standardization of PatientsLikeMe PGHD enables electronic accessibility and enhances patient engagement. Objective: The aim of this study is to retrospectively review PGHD for symptoms and ADEs entered by patients on PatientsLikeMe and coded by PatientsLikeMe to MedDRA terminology for concordance with regulatory-focused coding practices. Methods: An FDA MedDRA coding expert retrospectively reviewed a data file containing verbatim patient-reported symptoms and ADEs and PatientsLikeMe-assigned MedDRA terms to determine the medical accuracy and appropriateness of the selected MedDRA terms, applying the International Council for Harmonisation MedDRA Term Selection: Points to Consider (MTS:PTC) guides. Results: The FDA MedDRA coding expert reviewed 3234 PatientsLikeMe-assigned MedDRA codes and patient-reported verbatim text. The FDA and PatientsLikeMe were concordant at 97.09% (3140/3234) of the PatientsLikeMe-assigned MedDRA codes. The 2.91% (94/3234) discordant subset was analyzed to identify reasons for differences. Coding differences were attributed to several reasons but mostly driven by PatientsLikeMe?s approach of assigning a more general MedDRA term to enable patient-to-patient engagement, while the FDA assigned a more specific medically relevant term. Conclusions: PatientsLikeMe MedDRA coding of PGHD was generally comparable to how the FDA would code similar data, applying the MTS:PTC principles. Discordant coding resulted from several reasons but mostly reflected a difference in purpose. The MTS:PTC coding principles aim to capture the most specific reported information about an ADE, whereas PatientsLikeMe may code patient-reported symptoms and ADEs to more general MedDRA terms to support patient engagement among a larger group of patients. This study demonstrates that most verbatim reports of symptoms and ADEs collected by a PGHD source, such as the PatientsLikeMe platform, could be reliably coded to MedDRA terminology by applying the MTS:PTC guide. Regarding all secondary use of novel data, understanding coding and standardization principles applied to these data types are important. UR - http://medinform.jmir.org/2018/3/e42/ UR - http://dx.doi.org/10.2196/medinform.9878 UR - http://www.ncbi.nlm.nih.gov/pubmed/30131314 ID - info:doi/10.2196/medinform.9878 ER - TY - JOUR AU - Li, Jia AU - Liu, Minghui AU - Li, Xiaojun AU - Liu, Xuan AU - Liu, Jingfang PY - 2018/08/16 TI - Developing Embedded Taxonomy and Mining Patients? Interests From Web-Based Physician Reviews: Mixed-Methods Approach JO - J Med Internet Res SP - e254 VL - 20 IS - 8 KW - labeled-LDA KW - physicians KW - topic modeling KW - topic taxonomy KW - Web-based review N2 - Background: Web-based physician reviews are invaluable gold mines that merit further investigation. Although many studies have explored the text information of physician reviews, very few have focused on developing a systematic topic taxonomy embedded in physician reviews. The first step toward mining physician reviews is to determine how the natural structure or dimensions is embedded in reviews. Therefore, it is relevant to develop the topic taxonomy rigorously and systematically. Objective: This study aims to develop a hierarchical topic taxonomy to uncover the latent structure of physician reviews and illustrate its application for mining patients? interests based on the proposed taxonomy and algorithm. Methods: Data comprised 122,716 physician reviews, including reviews of 8501 doctors from a leading physician review website in China (haodf.com), collected between 2007 and 2015. Mixed methods, including a literature review, data-driven-based topic discovery, and human annotation were used to develop the physician review topic taxonomy. Results: The identified taxonomy included 3 domains or high-level categories and 9 subtopics or low-level categories. The physician-related domain included the categories of medical ethics, medical competence, communication skills, medical advice, and prescriptions. The patient-related domain included the categories of the patient profile, symptoms, diagnosis, and pathogenesis. The system-related domain included the categories of financing and operation process. The F-measure of the proposed classification algorithm reached 0.816 on average. Symptoms (Cohen d=1.58, ?u=0.216, t=229.75, and P<.001) are more often mentioned by patients with acute diseases, whereas communication skills (Cohen d=?0.29, ?u=?0.038, t=?42.01, and P<.001), financing (Cohen d=?0.68, ?u=?0.098, t=?99.26, and P<.001), and diagnosis and pathogenesis (Cohen d=?0.55, ?u=?0.078, t=?80.09, and P<.001) are more often mentioned by patients with chronic diseases. Patients with mild diseases were more interested in medical ethics (Cohen d=0.25, ?u 0.039, t=8.33, and P<.001), operation process (Cohen d=0.57, ?u 0.060, t=18.75, and P<.001), patient profile (Cohen d=1.19, ?u 0.132, t=39.33, and P<.001), and symptoms (Cohen d=1.91, ?u=0.274, t=62.82, and P<.001). Meanwhile, patients with serious diseases were more interested in medical competence (Cohen d=?0.99, ?u=?0.165, t=?32.58, and P<.001), medical advice and prescription (Cohen d=?0.65, ?u=?0.082, t=?21.45, and P<.001), financing (Cohen d=?0.26, ?u=?0.018, t=?8.45, and P<.001), and diagnosis and pathogenesis (Cohen d=?1.55, ?u=?0.229, t=?50.93, and P<.001). Conclusions: This mixed-methods approach, integrating literature reviews, data-driven topic discovery, and human annotation, is an effective and rigorous way to develop a physician review topic taxonomy. The proposed algorithm based on Labeled-Latent Dirichlet Allocation can achieve impressive classification results for mining patients? interests. Furthermore, the mining results reveal marked differences in patients? interests across different disease types, socioeconomic development levels, and hospital levels. UR - http://www.jmir.org/2018/8/e254/ UR - http://dx.doi.org/10.2196/jmir.8868 UR - http://www.ncbi.nlm.nih.gov/pubmed/30115610 ID - info:doi/10.2196/jmir.8868 ER - TY - JOUR AU - Al Manir, Sadnan Mohammad AU - Brenas, Haël Jon AU - Baker, JO Christopher AU - Shaban-Nejad, Arash PY - 2018/06/15 TI - A Surveillance Infrastructure for Malaria Analytics: Provisioning Data Access and Preservation of Interoperability JO - JMIR Public Health Surveill SP - e10218 VL - 4 IS - 2 KW - malaria surveillance KW - global health KW - interoperability KW - change management KW - Web services KW - population health intelligence N2 - Background: According to the World Health Organization, malaria surveillance is weakest in countries and regions with the highest malaria burden. A core obstacle is that the data required to perform malaria surveillance are fragmented in multiple data silos distributed across geographic regions. Furthermore, consistent integrated malaria data sources are few, and a low degree of interoperability exists between them. As a result, it is difficult to identify disease trends and to plan for effective interventions. Objective: We propose the Semantics, Interoperability, and Evolution for Malaria Analytics (SIEMA) platform for use in malaria surveillance based on semantic data federation. Using this approach, it is possible to access distributed data, extend and preserve interoperability between multiple dynamic distributed malaria sources, and facilitate detection of system changes that can interrupt mission-critical global surveillance activities. Methods: We used Semantic Automated Discovery and Integration (SADI) Semantic Web Services to enable data access and improve interoperability, and the graphical user interface-enabled semantic query engine HYDRA to implement the target queries typical of malaria programs. We implemented a custom algorithm to detect changes to community-developed terminologies, data sources, and services that are core to SIEMA. This algorithm reports to a dashboard. Valet SADI is used to mitigate the impact of changes by rebuilding affected services. Results: We developed a prototype surveillance and change management platform from a combination of third-party tools, community-developed terminologies, and custom algorithms. We illustrated a methodology and core infrastructure to facilitate interoperable access to distributed data sources using SADI Semantic Web services. This degree of access makes it possible to implement complex queries needed by our user community with minimal technical skill. We implemented a dashboard that reports on terminology changes that can render the services inactive, jeopardizing system interoperability. Using this information, end users can control and reactively rebuild services to preserve interoperability and minimize service downtime. Conclusions: We introduce a framework suitable for use in malaria surveillance that supports the creation of flexible surveillance queries across distributed data resources. The platform provides interoperable access to target data sources, is domain agnostic, and with updates to core terminological resources is readily transferable to other surveillance activities. A dashboard enables users to review changes to the infrastructure and invoke system updates. The platform significantly extends the range of functionalities offered by malaria information systems, beyond the state-of-the-art. UR - http://publichealth.jmir.org/2018/2/e10218/ UR - http://dx.doi.org/10.2196/10218 UR - http://www.ncbi.nlm.nih.gov/pubmed/29907554 ID - info:doi/10.2196/10218 ER - TY - JOUR AU - Merlo, Gianluca AU - Chiazzese, Giuseppe AU - Taibi, Davide AU - Chifari, Antonella PY - 2018/05/31 TI - Development and Validation of a Functional Behavioural Assessment Ontology to Support Behavioural Health Interventions JO - JMIR Med Inform SP - e37 VL - 6 IS - 2 KW - ontology KW - behavioral interventions KW - functional behavioral assessment KW - eHealth care KW - evidence-based practice N2 - Background: In the cognitive-behavioral approach, Functional Behavioural Assessment is one of the most effective methods to identify the variables that determine a problem behavior. In this context, the use of modern technologies can encourage the collection and sharing of behavioral patterns, effective intervention strategies, and statistical evidence about antecedents and consequences of clusters of problem behaviors, encouraging the designing of function-based interventions. Objective: The paper describes the development and validation process used to design a specific Functional Behavioural Assessment Ontology (FBA-Ontology). The FBA-Ontology is a semantic representation of the variables that intervene in a behavioral observation process, facilitating the systematic collection of behavioral data, the consequential planning of treatment strategies and, indirectly, the scientific advancement in this field of study. Methods: The ontology has been developed deducing concepts and relationships of the ontology from a gold standard and then performing a machine-based validation and a human-based assessment to validate the Functional Behavioural Assessment Ontology. These validation and verification processes were aimed to verify how much the ontology is conceptually well founded and semantically and syntactically correct. Results: The Pellet reasoner checked the logical consistency and the integrity of classes and properties defined in the ontology, not detecting any violation of constraints in the ontology definition. To assess whether the ontology definition is coherent with the knowledge domain, human evaluation of the ontology was performed asking 84 people to fill in a questionnaire composed by 13 questions assessing concepts, relations between concepts, and concepts? attributes. The response rate for the survey was 29/84 (34.52%). The domain experts confirmed that the concepts, the attributes, and the relationships between concepts defined in the FBA-Ontology are valid and well represent the Functional Behavioural Assessment process. Conclusions: The new ontology developed could be a useful tool to design new evidence-based systems in the Behavioral Interventions practices, encouraging the link with other Linked Open Data datasets and repositories to provide users with new models of eHealth focused on the management of problem behaviors. Therefore, new research is needed to develop and implement innovative strategies to improve the poor reproducibility and translatability of basic research findings in the field of behavioral assessment. UR - http://medinform.jmir.org/2018/2/e37/ UR - http://dx.doi.org/10.2196/medinform.7799 UR - http://www.ncbi.nlm.nih.gov/pubmed/ ID - info:doi/10.2196/medinform.7799 ER - TY - JOUR PY - 2018// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e8359 VL - 10 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v10i1.8359 ID - info:doi/10.5210/ojphi.v10i1.8359 ER - TY - JOUR AU - Chen, W. Henry AU - Du, Jingcheng AU - Song, Hsing-Yi AU - Liu, Xiangyu AU - Jiang, Guoqian AU - Tao, Cui PY - 2018/02/22 TI - Representation of Time-Relevant Common Data Elements in the Cancer Data Standards Repository: Statistical Evaluation of an Ontological Approach JO - JMIR Med Inform SP - e7 VL - 6 IS - 1 KW - common data elements KW - database management systems KW - database KW - time KW - biomedical ontology N2 - Background: Today, there is an increasing need to centralize and standardize electronic health data within clinical research as the volume of data continues to balloon. Domain-specific common data elements (CDEs) are emerging as a standard approach to clinical research data capturing and reporting. Recent efforts to standardize clinical study CDEs have been of great benefit in facilitating data integration and data sharing. The importance of the temporal dimension of clinical research studies has been well recognized; however, very few studies have focused on the formal representation of temporal constraints and temporal relationships within clinical research data in the biomedical research community. In particular, temporal information can be extremely powerful to enable high-quality cancer research. Objective: The objective of the study was to develop and evaluate an ontological approach to represent the temporal aspects of cancer study CDEs. Methods: We used CDEs recorded in the National Cancer Institute (NCI) Cancer Data Standards Repository (caDSR) and created a CDE parser to extract time-relevant CDEs from the caDSR. Using the Web Ontology Language (OWL)?based Time Event Ontology (TEO), we manually derived representative patterns to semantically model the temporal components of the CDEs using an observing set of randomly selected time-related CDEs (n=600) to create a set of TEO ontological representation patterns. In evaluating TEO?s ability to represent the temporal components of the CDEs, this set of representation patterns was tested against two test sets of randomly selected time-related CDEs (n=425). Results: It was found that 94.2% (801/850) of the CDEs in the test sets could be represented by the TEO representation patterns. Conclusions: In conclusion, TEO is a good ontological model for representing the temporal components of the CDEs recorded in caDSR. Our representative model can harness the Semantic Web reasoning and inferencing functionalities and present a means for temporal CDEs to be machine-readable, streamlining meaningful searches. UR - http://medinform.jmir.org/2018/1/e7/ UR - http://dx.doi.org/10.2196/medinform.8175 UR - http://www.ncbi.nlm.nih.gov/pubmed/29472179 ID - info:doi/10.2196/medinform.8175 ER - TY - JOUR AU - Chen, Jinying AU - Druhl, Emily AU - Polepalli Ramesh, Balaji AU - Houston, K. Thomas AU - Brandt, A. Cynthia AU - Zulman, M. Donna AU - Vimalananda, G. Varsha AU - Malkani, Samir AU - Yu, Hong PY - 2018/01/22 TI - A Natural Language Processing System That Links Medical Terms in Electronic Health Record Notes to Lay Definitions: System Development Using Physician Reviews JO - J Med Internet Res SP - e26 VL - 20 IS - 1 KW - electronic health records KW - natural language processing KW - consumer health informatics KW - usability testing KW - computer software N2 - Background: Many health care systems now allow patients to access their electronic health record (EHR) notes online through patient portals. Medical jargon in EHR notes can confuse patients, which may interfere with potential benefits of patient access to EHR notes. Objective: The aim of this study was to develop and evaluate the usability and content quality of NoteAid, a Web-based natural language processing system that links medical terms in EHR notes to lay definitions, that is, definitions easily understood by lay people. Methods: NoteAid incorporates two core components: CoDeMed, a lexical resource of lay definitions for medical terms, and MedLink, a computational unit that links medical terms to lay definitions. We developed innovative computational methods, including an adapted distant supervision algorithm to prioritize medical terms important for EHR comprehension to facilitate the effort of building CoDeMed. Ten physician domain experts evaluated the user interface and content quality of NoteAid. The evaluation protocol included a cognitive walkthrough session and a postsession questionnaire. Physician feedback sessions were audio-recorded. We used standard content analysis methods to analyze qualitative data from these sessions. Results: Physician feedback was mixed. Positive feedback on NoteAid included (1) Easy to use, (2) Good visual display, (3) Satisfactory system speed, and (4) Adequate lay definitions. Opportunities for improvement arising from evaluation sessions and feedback included (1) improving the display of definitions for partially matched terms, (2) including more medical terms in CoDeMed, (3) improving the handling of terms whose definitions vary depending on different contexts, and (4) standardizing the scope of definitions for medicines. On the basis of these results, we have improved NoteAid?s user interface and a number of definitions, and added 4502 more definitions in CoDeMed. Conclusions: Physician evaluation yielded useful feedback for content validation and refinement of this innovative tool that has the potential to improve patient EHR comprehension and experience using patient portals. Future ongoing work will develop algorithms to handle ambiguous medical terms and test and evaluate NoteAid with patients. UR - http://www.jmir.org/2018/1/e26/ UR - http://dx.doi.org/10.2196/jmir.8669 UR - http://www.ncbi.nlm.nih.gov/pubmed/29358159 ID - info:doi/10.2196/jmir.8669 ER - TY - JOUR AU - Qenam, Basel AU - Kim, Youn Tae AU - Carroll, J. Mark AU - Hogarth, Michael PY - 2017/12/18 TI - Text Simplification Using Consumer Health Vocabulary to Generate Patient-Centered Radiology Reporting: Translation and Evaluation JO - J Med Internet Res SP - e417 VL - 19 IS - 12 KW - consumer health information KW - vocabulary KW - radiology KW - electronic health records KW - natural language processing N2 - Background: Radiology reporting is a clinically oriented form of documentation that reflects critical information for patients about their health care processes. Realizing its importance, many medical institutions have started providing radiology reports in patient portals. The gain, however, can be limited because of medical language barriers, which require a way for customizing these reports for patients. The open-access, collaborative consumer health vocabulary (CHV) is a terminology system created for such purposes and can be the basis of lexical simplification processes for clinical notes. Objective: The aim of this study was to examine the comprehensibility and suitability of CHV in simplifying radiology reports for consumers. This was done by characterizing the content coverage and the lexical similarity between the terms in the reports and the CHV-preferred terms. Methods: The overall procedure was divided into the following two main stages: (1) translation and (2) evaluation. The translation process involved using MetaMap to link terms in the reports to CHV concepts. This is followed by replacing the terms with CHV-preferred terms using the concept names and sources table (MRCONSO) in the Unified Medical Language System (UMLS) Metathesaurus. In the second stage, medical terms in the reports and general terms that are used to describe medical phenomena were selected and evaluated by comparing the words in the original reports with the translated ones. The evaluation includes measuring the content coverage, investigating lexical similarity, and finding trends in missing concepts. Results: Of the 792 terms selected from the radiology reports, 695 of them could be mapped directly to CHV concepts, indicating a content coverage of 88.5%. A total of 51 of the concepts (53%, 51/97) that could not be mapped are names of human anatomical structures and regions, followed by 28 anatomical descriptions and pathological variations (29%, 28/97). In addition, 12 radiology techniques and projections represented 12% of the unmapped concepts, whereas the remaining six concepts (6%, 12/97) were physiological descriptions. The rate of lexical similarity between the CHV-preferred terms and the terms in the radiology reports was approximately 72.6%. Conclusions: The CHV covered a high percentage of concepts found in the radiology reports, but unmapped concepts are associated with areas that are commonly found in radiology reporting. CHV terms also showed a high percentage of lexical similarity with terms in the reports, which contain a myriad of medical jargon. This suggests that many CHV terms might not be suitable for lay consumers who would not be facile with radiology-specific vocabulary. Therefore, further patient-centered content changes are needed of the CHV to increase its usefulness and facilitate its integration into consumer-oriented applications. UR - http://www.jmir.org/2017/12/e417/ UR - http://dx.doi.org/10.2196/jmir.8536 UR - http://www.ncbi.nlm.nih.gov/pubmed/29254915 ID - info:doi/10.2196/jmir.8536 ER - TY - JOUR AU - Beitia, Oscar Anton AU - Lowry, Tina AU - Vreeman, J. Daniel AU - Loo, T. George AU - Delman, N. Bradley AU - Thum, L. Frederick AU - Slovis, H. Benjamin AU - Shapiro, S. Jason PY - 2017/12/14 TI - Standard Anatomic Terminologies: Comparison for Use in a Health Information Exchange?Based Prior Computed Tomography (CT) Alerting System JO - JMIR Med Inform SP - e49 VL - 5 IS - 4 KW - tomography, x-ray computed KW - health information exchange KW - radiation dosage KW - terminology KW - anatomy, regional N2 - Background: A health information exchange (HIE)?based prior computed tomography (CT) alerting system may reduce avoidable CT imaging by notifying ordering clinicians of prior relevant studies when a study is ordered. For maximal effectiveness, a system would alert not only for prior same CTs (exams mapped to the same code from an exam name terminology) but also for similar CTs (exams mapped to different exam name terminology codes but in the same anatomic region) and anatomically proximate CTs (exams in adjacent anatomic regions). Notification of previous same studies across an HIE requires mapping of local site CT codes to a standard terminology for exam names (such as Logical Observation Identifiers Names and Codes [LOINC]) to show that two studies with different local codes and descriptions are equivalent. Notifying of prior similar or proximate CTs requires an additional mapping of exam codes to anatomic regions, ideally coded by an anatomic terminology. Several anatomic terminologies exist, but no prior studies have evaluated how well they would support an alerting use case. Objective: The aim of this study was to evaluate the fitness of five existing standard anatomic terminologies to support similar or proximate alerts of an HIE-based prior CT alerting system. Methods: We compared five standard anatomic terminologies (Foundational Model of Anatomy, Systematized Nomenclature of Medicine Clinical Terms, RadLex, LOINC, and LOINC/Radiological Society of North America [RSNA] Radiology Playbook) to an anatomic framework created specifically for our use case (Simple ANatomic Ontology for Proximity or Similarity [SANOPS]), to determine whether the existing terminologies could support our use case without modification. On the basis of an assessment of optimal terminology features for our purpose, we developed an ordinal anatomic terminology utility classification. We mapped samples of 100 random and the 100 most frequent LOINC CT codes to anatomic regions in each terminology, assigned utility classes for each mapping, and statistically compared each terminology?s utility class rankings. We also constructed seven hypothetical alerting scenarios to illustrate the terminologies? differences. Results: Both RadLex and the LOINC/RSNA Radiology Playbook anatomic terminologies ranked significantly better (P<.001) than the other standard terminologies for the 100 most frequent CTs, but no terminology ranked significantly better than any other for 100 random CTs. Hypothetical scenarios illustrated instances where no standard terminology would support appropriate proximate or similar alerts, without modification. Conclusions: LOINC/RSNA Radiology Playbook and RadLex?s anatomic terminologies appear well suited to support proximate or similar alerts for commonly ordered CTs, but for less commonly ordered tests, modification of the existing terminologies with concepts and relations from SANOPS would likely be required. Our findings suggest SANOPS may serve as a framework for enhancing anatomic terminologies in support of other similar use cases. UR - http://medinform.jmir.org/2017/4/e49/ UR - http://dx.doi.org/10.2196/medinform.8765 UR - http://www.ncbi.nlm.nih.gov/pubmed/29242174 ID - info:doi/10.2196/medinform.8765 ER - TY - JOUR AU - Chen, Jinying AU - Jagannatha, N. Abhyuday AU - Fodeh, J. Samah AU - Yu, Hong PY - 2017/10/31 TI - Ranking Medical Terms to Support Expansion of Lay Language Resources for Patient Comprehension of Electronic Health Record Notes: Adapted Distant Supervision Approach JO - JMIR Med Inform SP - e42 VL - 5 IS - 4 KW - electronic health records KW - natural language processing KW - lexical entry selection KW - transfer learning KW - information extraction N2 - Background: Medical terms are a major obstacle for patients to comprehend their electronic health record (EHR) notes. Clinical natural language processing (NLP) systems that link EHR terms to lay terms or definitions allow patients to easily access helpful information when reading through their EHR notes, and have shown to improve patient EHR comprehension. However, high-quality lay language resources for EHR terms are very limited in the public domain. Because expanding and curating such a resource is a costly process, it is beneficial and even necessary to identify terms important for patient EHR comprehension first. Objective: We aimed to develop an NLP system, called adapted distant supervision (ADS), to rank candidate terms mined from EHR corpora. We will give EHR terms ranked as high by ADS a higher priority for lay language annotation?that is, creating lay definitions for these terms. Methods: Adapted distant supervision uses distant supervision from consumer health vocabulary and transfer learning to adapt itself to solve the problem of ranking EHR terms in the target domain. We investigated 2 state-of-the-art transfer learning algorithms (ie, feature space augmentation and supervised distant supervision) and designed 5 types of learning features, including distributed word representations learned from large EHR data for ADS. For evaluating ADS, we asked domain experts to annotate 6038 candidate terms as important or nonimportant for EHR comprehension. We then randomly divided these data into the target-domain training data (1000 examples) and the evaluation data (5038 examples). We compared ADS with 2 strong baselines, including standard supervised learning, on the evaluation data. Results: The ADS system using feature space augmentation achieved the best average precision, 0.850, on the evaluation set when using 1000 target-domain training examples. The ADS system using supervised distant supervision achieved the best average precision, 0.819, on the evaluation set when using only 100 target-domain training examples. The 2 ADS systems both performed significantly better than the baseline systems (P<.001 for all measures and all conditions). Using a rich set of learning features contributed to ADS?s performance substantially. Conclusions: ADS can effectively rank terms mined from EHRs. Transfer learning improved ADS?s performance even with a small number of target-domain training examples. EHR terms prioritized by ADS were used to expand a lay language resource that supports patient EHR comprehension. The top 10,000 EHR terms ranked by ADS are available upon request. UR - http://medinform.jmir.org/2017/4/e42/ UR - http://dx.doi.org/10.2196/medinform.8531 UR - http://www.ncbi.nlm.nih.gov/pubmed/29089288 ID - info:doi/10.2196/medinform.8531 ER - TY - JOUR AU - Zhang, Zhizun AU - Gonzalez, C. Mila AU - Morse, S. Stephen AU - Venkatasubramanian, Venkat PY - 2017/10/11 TI - Knowledge Management Framework for Emerging Infectious Diseases Preparedness and Response: Design and Development of Public Health Document Ontology JO - JMIR Res Protoc SP - e196 VL - 6 IS - 10 KW - EIDs KW - public health KW - systems engineering KW - knowledge representation KW - teleological function KW - knowledge management KW - ontology KW - semantic reasoning N2 - Background: There are increasing concerns about our preparedness and timely coordinated response across the globe to cope with emerging infectious diseases (EIDs). This poses practical challenges that require exploiting novel knowledge management approaches effectively. Objective: This work aims to develop an ontology-driven knowledge management framework that addresses the existing challenges in sharing and reusing public health knowledge. Methods: We propose a systems engineering-inspired ontology-driven knowledge management approach. It decomposes public health knowledge into concepts and relations and organizes the elements of knowledge based on the teleological functions. Both knowledge and semantic rules are stored in an ontology and retrieved to answer queries regarding EID preparedness and response. Results: A hybrid concept extraction was implemented in this work. The quality of the ontology was evaluated using the formal evaluation method Ontology Quality Evaluation Framework. Conclusions: Our approach is a potentially effective methodology for managing public health knowledge. Accuracy and comprehensiveness of the ontology can be improved as more knowledge is stored. In the future, a survey will be conducted to collect queries from public health practitioners. The reasoning capacity of the ontology will be evaluated using the queries and hypothetical outbreaks. We suggest the importance of developing a knowledge sharing standard like the Gene Ontology for the public health domain. UR - http://www.researchprotocols.org/2017/10/e196/ UR - http://dx.doi.org/10.2196/resprot.7904 UR - http://www.ncbi.nlm.nih.gov/pubmed/29021130 ID - info:doi/10.2196/resprot.7904 ER - TY - JOUR AU - de Lusignan, Simon AU - Shinneman, Stacy AU - Yonova, Ivelina AU - van Vlymen, Jeremy AU - Elliot, J. Alex AU - Bolton, Frederick AU - Smith, E. Gillian AU - O'Brien, Sarah PY - 2017/09/28 TI - An Ontology to Improve Transparency in Case Definition and Increase Case Finding of Infectious Intestinal Disease: Database Study in English General Practice JO - JMIR Med Inform SP - e34 VL - 5 IS - 3 KW - dysentery KW - enteritis KW - enterobacteriaceae KW - enterocolitis KW - gastritis KW - gastroenteritis KW - intestinal diseases KW - medical records systems, computerized KW - norovirus KW - primary health care N2 - Background: Infectious intestinal disease (IID) has considerable health impact; there are 2 billion cases worldwide resulting in 1 million deaths and 78.7 million disability-adjusted life years lost. Reported IID incidence rates vary and this is partly because terms such as ?diarrheal disease? and ?acute infectious gastroenteritis? are used interchangeably. Ontologies provide a method of transparently comparing case definitions and disease incidence rates. Objective: This study sought to show how differences in case definition in part account for variation in incidence estimates for IID and how an ontological approach provides greater transparency to IID case finding. Methods: We compared three IID case definitions: (1) Royal College of General Practitioners Research and Surveillance Centre (RCGP RSC) definition based on mapping to the Ninth International Classification of Disease (ICD-9), (2) newer ICD-10 definition, and (3) ontological case definition. We calculated incidence rates and examined the contribution of four supporting concepts related to IID: symptoms, investigations, process of care (eg, notification to public health authorities), and therapies. We created a formal ontology using ontology Web language. Results: The ontological approach identified 5712 more cases of IID than the ICD-10 definition and 4482 more than the RCGP RSC definition from an initial cohort of 1,120,490. Weekly incidence using the ontological definition was 17.93/100,000 (95% CI 15.63-20.41), whereas for the ICD-10 definition the rate was 8.13/100,000 (95% CI 6.70-9.87), and for the RSC definition the rate was 10.24/100,000 (95% CI 8.55-12.12). Codes from the four supporting concepts were generally consistent across our three IID case definitions: 37.38% (3905/10,448) (95% CI 36.16-38.5) for the ontological definition, 38.33% (2287/5966) (95% CI 36.79-39.93) for the RSC definition, and 40.82% (1933/4736) (95% CI 39.03-42.66) for the ICD-10 definition. The proportion of laboratory results associated with a positive test result was 19.68% (546/2775). Conclusions: The standard RCGP RSC definition of IID, and its mapping to ICD-10, underestimates disease incidence. The ontological approach identified a larger proportion of new IID cases; the ontology divides contributory elements and enables transparency and comparison of rates. Results illustrate how improved diagnostic coding of IID combined with an ontological approach to case definition would provide a clearer picture of IID in the community, better inform GPs and public health services about circulating disease, and empower them to respond. We need to improve the Pathology Bounded Code List (PBCL) currently used by laboratories to electronically report results. Given advances in stool microbiology testing with a move to nonculture, PCR-based methods, the way microbiology results are reported and coded via PBCL needs to be reviewed and modernized. UR - http://medinform.jmir.org/2017/3/e34/ UR - http://dx.doi.org/10.2196/medinform.7641 UR - http://www.ncbi.nlm.nih.gov/pubmed/28958989 ID - info:doi/10.2196/medinform.7641 ER - TY - JOUR AU - Chen, T. Annie AU - Carriere, M. Rachel AU - Kaplan, Jan Samantha PY - 2017/09/07 TI - The User Knows What to Call It: Incorporating Patient Voice Through User-Contributed Tags on a Participatory Platform About Health Management JO - J Med Internet Res SP - e292 VL - 19 IS - 9 KW - collaborative tagging KW - folksonomy KW - knowledge organization KW - self-management KW - body listening KW - body awareness N2 - Background: Body listening, described as the act of paying attention to the body?s signals and cues, can be an important component of long-term health management. Objective: The aim of this study was to introduce and evaluate the Body Listening Project, an innovative effort to engage the public in the creation of a public resource?to leverage collective wisdom in the health domain. This project involved a website where people could contribute their experiences of and dialogue with others concerning body listening and self-management. This article presents an analysis of the tags contributed, with a focus on the value of these tags for knowledge organization and incorporation into consumer-friendly health information retrieval systems. Methods: First, we performed content analysis of the tags contributed, identifying a set of categories and refining the relational structure of the categories to develop a preliminary classification scheme, the Body Listening and Self-Management Taxonomy. Second, we compared the concepts in the Body Listening and Self-Management Taxonomy with concepts that were automatically identified from an extant health knowledge resource, the Unified Medical Language System (UMLS), to better characterize the information that participants contributed. Third, we employed visualization techniques to explore the concept space of the tags. A correlation matrix, based on the extent to which categories tended to be assigned to the same tags, was used to study the interrelatedness of the taxonomy categories. Then a network visualization was used to investigate structural relationships among the categories in the taxonomy. Results: First, we proposed a taxonomy called the Body Listening and Self-Management Taxonomy, with four meta-level categories: (1) health management strategies, (2) concepts and states, (3) influencers, and (4) health-related information behavior. This taxonomy could inform future efforts to organize knowledge and content of this subject matter. Second, we compared the categories from this taxonomy with the UMLS concepts that were identified. Though the UMLS offers benefits such as speed and breadth of coverage, the Body Listening and Self-Management Taxonomy is more consumer-centric. Third, the correlation matrix and network visualization demonstrated that there are natural areas of ambiguity and semantic relatedness in the meanings of the concepts in the Body Listening and Self-Management Taxonomy. Use of these visualizations can be helpful in practice settings, to help library and information science practitioners understand and resolve potential challenges in classification; in research, to characterize the structure of the conceptual space of health management; and in the development of consumer-centric health information retrieval systems. Conclusions: A participatory platform can be employed to collect data concerning patient experiences of health management, which can in turn be used to develop new health knowledge resources or augment existing ones, as well as be incorporated into consumer-centric health information systems. UR - http://www.jmir.org/2017/9/e292/ UR - http://dx.doi.org/10.2196/jmir.7673 UR - http://www.ncbi.nlm.nih.gov/pubmed/28882809 ID - info:doi/10.2196/jmir.7673 ER - TY - JOUR AU - Jung, Hyesil AU - Park, Hyeoun-Ae AU - Song, Tae-Min PY - 2017/07/24 TI - Ontology-Based Approach to Social Data Sentiment Analysis: Detection of Adolescent Depression Signals JO - J Med Internet Res SP - e259 VL - 19 IS - 7 KW - ontology KW - adolescent KW - depression KW - data mining KW - social media data N2 - Background: Social networking services (SNSs) contain abundant information about the feelings, thoughts, interests, and patterns of behavior of adolescents that can be obtained by analyzing SNS postings. An ontology that expresses the shared concepts and their relationships in a specific field could be used as a semantic framework for social media data analytics. Objective: The aim of this study was to refine an adolescent depression ontology and terminology as a framework for analyzing social media data and to evaluate description logics between classes and the applicability of this ontology to sentiment analysis. Methods: The domain and scope of the ontology were defined using competency questions. The concepts constituting the ontology and terminology were collected from clinical practice guidelines, the literature, and social media postings on adolescent depression. Class concepts, their hierarchy, and the relationships among class concepts were defined. An internal structure of the ontology was designed using the entity-attribute-value (EAV) triplet data model, and superclasses of the ontology were aligned with the upper ontology. Description logics between classes were evaluated by mapping concepts extracted from the answers to frequently asked questions (FAQs) onto the ontology concepts derived from description logic queries. The applicability of the ontology was validated by examining the representability of 1358 sentiment phrases using the ontology EAV model and conducting sentiment analyses of social media data using ontology class concepts. Results: We developed an adolescent depression ontology that comprised 443 classes and 60 relationships among the classes; the terminology comprised 1682 synonyms of the 443 classes. In the description logics test, no error in relationships between classes was found, and about 89% (55/62) of the concepts cited in the answers to FAQs mapped onto the ontology class. Regarding applicability, the EAV triplet models of the ontology class represented about 91.4% of the sentiment phrases included in the sentiment dictionary. In the sentiment analyses, ?academic stresses? and ?suicide? contributed negatively to the sentiment of adolescent depression. Conclusions: The ontology and terminology developed in this study provide a semantic foundation for analyzing social media data on adolescent depression. To be useful in social media data analysis, the ontology, especially the terminology, needs to be updated constantly to reflect rapidly changing terms used by adolescents in social media postings. In addition, more attributes and value sets reflecting depression-related sentiments should be added to the ontology. UR - http://www.jmir.org/2017/7/e259/ UR - http://dx.doi.org/10.2196/jmir.7452 UR - http://www.ncbi.nlm.nih.gov/pubmed/28739560 ID - info:doi/10.2196/jmir.7452 ER - TY - JOUR AU - Zheng, Shuai AU - Lu, J. James AU - Ghasemzadeh, Nima AU - Hayek, S. Salim AU - Quyyumi, A. Arshed AU - Wang, Fusheng PY - 2017/05/09 TI - Effective Information Extraction Framework for Heterogeneous Clinical Reports Using Online Machine Learning and Controlled Vocabularies JO - JMIR Med Inform SP - e12 VL - 5 IS - 2 KW - information extraction KW - natural language processing KW - controlled vocabulary KW - electronic medical records N2 - Background: Extracting structured data from narrated medical reports is challenged by the complexity of heterogeneous structures and vocabularies and often requires significant manual effort. Traditional machine-based approaches lack the capability to take user feedbacks for improving the extraction algorithm in real time. Objective: Our goal was to provide a generic information extraction framework that can support diverse clinical reports and enables a dynamic interaction between a human and a machine that produces highly accurate results. Methods: A clinical information extraction system IDEAL-X has been built on top of online machine learning. It processes one document at a time, and user interactions are recorded as feedbacks to update the learning model in real time. The updated model is used to predict values for extraction in subsequent documents. Once prediction accuracy reaches a user-acceptable threshold, the remaining documents may be batch processed. A customizable controlled vocabulary may be used to support extraction. Results: Three datasets were used for experiments based on report styles: 100 cardiac catheterization procedure reports, 100 coronary angiographic reports, and 100 integrated reports?each combines history and physical report, discharge summary, outpatient clinic notes, outpatient clinic letter, and inpatient discharge medication report. Data extraction was performed by 3 methods: online machine learning, controlled vocabularies, and a combination of these. The system delivers results with F1 scores greater than 95%. Conclusions: IDEAL-X adopts a unique online machine learning?based approach combined with controlled vocabularies to support data extraction for clinical reports. The system can quickly learn and improve, thus it is highly adaptable. UR - http://medinform.jmir.org/2017/2/e12/ UR - http://dx.doi.org/10.2196/medinform.7235 UR - http://www.ncbi.nlm.nih.gov/pubmed/28487265 ID - info:doi/10.2196/medinform.7235 ER - TY - JOUR PY - 2017// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e7688 VL - 9 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v9i1.7688 ID - info:doi/10.5210/ojphi.v9i1.7688 ER - TY - JOUR AU - Demelo, Jonathan AU - Parsons, Paul AU - Sedig, Kamran PY - 2017/02/02 TI - Ontology-Driven Search and Triage: Design of a Web-Based Visual Interface for MEDLINE JO - JMIR Med Inform SP - e4 VL - 5 IS - 1 KW - MEDLINE KW - user-computer interface KW - information storage and retrieval KW - medical informatics KW - PubMed N2 - Background: Diverse users need to search health and medical literature to satisfy open-ended goals such as making evidence-based decisions and updating their knowledge. However, doing so is challenging due to at least two major difficulties: (1) articulating information needs using accurate vocabulary and (2) dealing with large document sets returned from searches. Common search interfaces such as PubMed do not provide adequate support for exploratory search tasks. Objective: Our objective was to improve support for exploratory search tasks by combining two strategies in the design of an interactive visual interface by (1) using a formal ontology to help users build domain-specific knowledge and vocabulary and (2) providing multi-stage triaging support to help mitigate the information overload problem. Methods: We developed a Web-based tool, Ontology-Driven Visual Search and Triage Interface for MEDLINE (OVERT-MED), to test our design ideas. We implemented a custom searchable index of MEDLINE, which comprises approximately 25 million document citations. We chose a popular biomedical ontology, the Human Phenotype Ontology (HPO), to test our solution to the vocabulary problem. We implemented multistage triaging support in OVERT-MED, with the aid of interactive visualization techniques, to help users deal with large document sets returned from searches. Results: Formative evaluation suggests that the design features in OVERT-MED are helpful in addressing the two major difficulties described above. Using a formal ontology seems to help users articulate their information needs with more accurate vocabulary. In addition, multistage triaging combined with interactive visualizations shows promise in mitigating the information overload problem. Conclusions: Our strategies appear to be valuable in addressing the two major problems in exploratory search. Although we tested OVERT-MED with a particular ontology and document collection, we anticipate that our strategies can be transferred successfully to other contexts. UR - http://medinform.jmir.org/2017/1/e4/ UR - http://dx.doi.org/10.2196/medinform.6918 UR - http://www.ncbi.nlm.nih.gov/pubmed/28153818 ID - info:doi/10.2196/medinform.6918 ER - TY - JOUR PY - 2016// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e7052 VL - 8 IS - 3 UR - UR - http://dx.doi.org/10.5210/ojphi.v8i3.7052 UR - http://www.ncbi.nlm.nih.gov/pubmed/28210417 ID - info:doi/10.5210/ojphi.v8i3.7052 ER - TY - JOUR AU - Eivazzadeh, Shahryar AU - Anderberg, Peter AU - Larsson, C. Tobias AU - Fricker, A. Samuel AU - Berglund, Johan PY - 2016/06/16 TI - Evaluating Health Information Systems Using Ontologies JO - JMIR Med Inform SP - e20 VL - 4 IS - 2 KW - health information systems KW - ontologies KW - evaluation KW - technology assessment KW - biomedical N2 - Background: There are several frameworks that attempt to address the challenges of evaluation of health information systems by offering models, methods, and guidelines about what to evaluate, how to evaluate, and how to report the evaluation results. Model-based evaluation frameworks usually suggest universally applicable evaluation aspects but do not consider case-specific aspects. On the other hand, evaluation frameworks that are case specific, by eliciting user requirements, limit their output to the evaluation aspects suggested by the users in the early phases of system development. In addition, these case-specific approaches extract different sets of evaluation aspects from each case, making it challenging to collectively compare, unify, or aggregate the evaluation of a set of heterogeneous health information systems. Objectives: The aim of this paper is to find a method capable of suggesting evaluation aspects for a set of one or more health information systems?whether similar or heterogeneous?by organizing, unifying, and aggregating the quality attributes extracted from those systems and from an external evaluation framework. Methods: On the basis of the available literature in semantic networks and ontologies, a method (called Unified eValuation using Ontology; UVON) was developed that can organize, unify, and aggregate the quality attributes of several health information systems into a tree-style ontology structure. The method was extended to integrate its generated ontology with the evaluation aspects suggested by model-based evaluation frameworks. An approach was developed to extract evaluation aspects from the ontology that also considers evaluation case practicalities such as the maximum number of evaluation aspects to be measured or their required degree of specificity. The method was applied and tested in Future Internet Social and Technological Alignment Research (FI-STAR), a project of 7 cloud-based eHealth applications that were developed and deployed across European Union countries. Results: The relevance of the evaluation aspects created by the UVON method for the FI-STAR project was validated by the corresponding stakeholders of each case. These evaluation aspects were extracted from a UVON-generated ontology structure that reflects both the internally declared required quality attributes in the 7 eHealth applications of the FI-STAR project and the evaluation aspects recommended by the Model for ASsessment of Telemedicine applications (MAST) evaluation framework. The extracted evaluation aspects were used to create questionnaires (for the corresponding patients and health professionals) to evaluate each individual case and the whole of the FI-STAR project. Conclusions: The UVON method can provide a relevant set of evaluation aspects for a heterogeneous set of health information systems by organizing, unifying, and aggregating the quality attributes through ontological structures. Those quality attributes can be either suggested by evaluation models or elicited from the stakeholders of those systems in the form of system requirements. The method continues to be systematic, context sensitive, and relevant across a heterogeneous set of health information systems. UR - http://medinform.jmir.org/2016/2/e20/ UR - http://dx.doi.org/10.2196/medinform.5185 UR - http://www.ncbi.nlm.nih.gov/pubmed/27311735 ID - info:doi/10.2196/medinform.5185 ER - TY - JOUR AU - Schmitz, Matthew AU - Forst, Linda PY - 2016/02/15 TI - Industry and Occupation in the Electronic Health Record: An Investigation of the National Institute for Occupational Safety and Health Industry and Occupation Computerized Coding System JO - JMIR Med Inform SP - e5 VL - 4 IS - 1 KW - medical informatics KW - occupation code KW - industry code KW - NIOCCS KW - occupational health KW - occupation+electronic health record N2 - Background: Inclusion of information about a patient?s work, industry, and occupation, in the electronic health record (EHR) could facilitate occupational health surveillance, better health outcomes, prevention activities, and identification of workers? compensation cases. The US National Institute for Occupational Safety and Health (NIOSH) has developed an autocoding system for ?industry? and ?occupation? based on 1990 Bureau of Census codes; its effectiveness requires evaluation in conjunction with promoting the mandatory addition of these variables to the EHR. Objective: The objective of the study was to evaluate the intercoder reliability of NIOSH?s Industry and Occupation Computerized Coding System (NIOCCS) when applied to data collected in a community survey conducted under the Affordable Care Act; to determine the proportion of records that are autocoded using NIOCCS. Methods: Standard Occupational Classification (SOC) codes are used by several federal agencies in databases that capture demographic, employment, and health information to harmonize variables related to work activities among these data sources. There are 359 industry and occupation responses that were hand coded by 2 investigators, who came to a consensus on every code. The same variables were autocoded using NIOCCS at the high and moderate criteria level. Results: Kappa was .84 for agreement between hand coders and between the hand coder consensus code versus NIOCCS high confidence level codes for the first 2 digits of the SOC code. For 4 digits, NIOCCS coding versus investigator coding ranged from kappa=.56 to .70. In this study, NIOCCS was able to achieve production rates (ie, to autocode) 31%-36% of entered variables at the ?high confidence? level and 49%-58% at the ?medium confidence? level. Autocoding (production) rates are somewhat lower than those reported by NIOSH. Agreement between manually coded and autocoded data are ?substantial? at the 2-digit level, but only ?fair? to ?good? at the 4-digit level. Conclusions: This work serves as a baseline for performance of NIOCCS by investigators in the field. Further field testing will clarify NIOCCS effectiveness in terms of ability to assign codes and coding accuracy and will clarify its value as inclusion of these occupational variables in the EHR is promoted. UR - http://medinform.jmir.org/2016/1/e5/ UR - http://dx.doi.org/10.2196/medinform.4839 UR - http://www.ncbi.nlm.nih.gov/pubmed/26878932 ID - info:doi/10.2196/medinform.4839 ER - TY - JOUR PY - 2015// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e6010 VL - 7 IS - 2 UR - UR - http://dx.doi.org/10.5210/ojphi.v7i2.6010 UR - http://www.ncbi.nlm.nih.gov/pubmed/26392849 ID - info:doi/10.5210/ojphi.v7i2.6010 ER - TY - JOUR AU - Albin, Aaron AU - Ji, Xiaonan AU - Borlawsky, B. Tara AU - Ye, Zhan AU - Lin, Simon AU - Payne, RO Philip AU - Huang, Kun AU - Xiang, Yang PY - 2014/10/07 TI - Enabling Online Studies of Conceptual Relationships Between Medical Terms: Developing an Efficient Web Platform JO - JMIR Med Inform SP - e23 VL - 2 IS - 2 KW - UMLS KW - ontology KW - conceptual relationships N2 - Background: The Unified Medical Language System (UMLS) contains many important ontologies in which terms are connected by semantic relations. For many studies on the relationships between biomedical concepts, the use of transitively associated information from ontologies and the UMLS has been shown to be effective. Although there are a few tools and methods available for extracting transitive relationships from the UMLS, they usually have major restrictions on the length of transitive relations or on the number of data sources. Objective: Our goal was to design an efficient online platform that enables efficient studies on the conceptual relationships between any medical terms. Methods: To overcome the restrictions of available methods and to facilitate studies on the conceptual relationships between medical terms, we developed a Web platform, onGrid, that supports efficient transitive queries and conceptual relationship studies using the UMLS. This framework uses the latest technique in converting natural language queries into UMLS concepts, performs efficient transitive queries, and visualizes the result paths. It also dynamically builds a relationship matrix for two sets of input biomedical terms. We are thus able to perform effective studies on conceptual relationships between medical terms based on their relationship matrix. Results: The advantage of onGrid is that it can be applied to study any two sets of biomedical concept relations and the relations within one set of biomedical concepts. We use onGrid to study the disease-disease relationships in the Online Mendelian Inheritance in Man (OMIM). By crossvalidating our results with an external database, the Comparative Toxicogenomics Database (CTD), we demonstrated that onGrid is effective for the study of conceptual relationships between medical terms. Conclusions: onGrid is an efficient tool for querying the UMLS for transitive relations, studying the relationship between medical terms, and generating hypotheses. UR - http://medinform.jmir.org/2014/2/e23/ UR - http://dx.doi.org/10.2196/medinform.3387 UR - http://www.ncbi.nlm.nih.gov/pubmed/25600290 ID - info:doi/10.2196/medinform.3387 ER - TY - JOUR PY - 2013// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e4440 VL - 5 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v5i1.4440 ID - info:doi/10.5210/ojphi.v5i1.4440 ER - TY - JOUR PY - 2013// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e4447 VL - 5 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v5i1.4447 ID - info:doi/10.5210/ojphi.v5i1.4447 ER - TY - JOUR PY - 2013// TI - Roles of Health Literacy in Relation to Social Determinants of Health and Recommendations for Informatics-Based Interventions: Systematic Review JO - Online J Public Health Inform SP - e4452 VL - 5 IS - 1 UR - UR - http://dx.doi.org/10.5210/ojphi.v5i1.4452 ID - info:doi/10.5210/ojphi.v5i1.4452 ER -